Building a TEI Release
NOTE: THIS DOCUMENT is currently in transition, as we move from keeping our code repository on SourceForge and releasing our products there to a process based on GitHub. It is therefore potentially incomplete.
This document aims to provide a set of detailed instructions enabling a "release technician" (the Council member tasked with implementing a new release of the TEI) to prepare for and manage the release process. It assumes that the new packages will be taken from one of the Jenkins servers rather than being built locally by the release technician. This is easier and more reliable, because we ensure that the Jenkins servers are regularly updated and are correctly configured to build the TEI products.
Packages on GitHub
The TEI maintains a number of distinct repository on GitHub, under the TEIC organization. The main repository for the developing P5 Guidelines and associated schemas is TEIC/TEI, and the TEI Stylesheets, the code for the Roma web application, and the source code for the Oxygen TEI plugin can be found in TEIC/Stylesheets, TEIC/Roma, and TEIC/oxygen-tei respectively.
The rest of this section describes how to make a new release for the main P5 package, but similar procedures apply to the others. The instructions assume you are working on a Linux or MacOSX system with a command line, and know (more or less) how to do basic command-line operations such as running scripts and logging into a server with ssh.
What you will need before you start
- An account on GitHub, and committer privileges over the TEI repository. If you have ever pushed a change to the master branch TEI repository, you should have all the required permissions.
- Shell access on the TEI SourceForge project. Contact one of the admins to have this turned on. Normal committers don't have shell access. NOTE: THIS IS ONLY REQUIRED IF WE CONTINUE TO RELEASE PRODUCTS THROUGH THE SOURCEFORGE WEBSITE. COUNCIL IS STILL IN THE PROCESS OF DECIDING WHETHER WE SHOULD CONTINUE TO DO THIS.
The release manager will need SSH login access to the tei account on the tei-c.org
server. This involves two steps:
- Generate an SSH key pair (if you don't have one already). If this is new to you, look at https://en.wikipedia.org/wiki/Ssh-keygen.
- Send the public key to the Council Chair, who will forward it on to the system administrator.
- Some familiarity with the two TEI Jenkins Continuous Integration Servers (their URLs are below). Take a little time to watch them work, and see how a push to the GitHub repository causes them to start building TEI packages. There are three specific build jobs associated with P5, and they run in a fixed sequence.
- A nickname for irc.freenode.net, so you can log into #tei-c channel for live help from other Council members. For more information, see the TEI wiki page on IRC.
- Several hours of time. You won't be busy all the time, but the process from beginning to end takes several hours, especially if something goes a bit wrong and you have to retrace your steps. It's best to start first thing in the morning, and prepare to be busy all day.
NOTE: THE FOLLOWING ITEMS ARE ONLY RELEVANT IF WE CONTINUE TO RELEASE PRODUCTS ON THE SOURCEFORGE WEBSITE.
A copy of the public key that will enable you to sync the release zip with
- log in to the tei server (ssh email@example.com; this requires that you've completed the other public key step above).
- cat ~/.ssh/id_rsa.pub and copy the contents to the clipboard.
- paste the result into a text editor and remove any linebreaks added by the terminal.
- copy-paste the result into https://sourceforge.net/account/ssh
Test it by trying to log into SourceForge via ssh from the tei-c.org server:
where "sfuser" is your SourceForge user name. You should not see a prompt for a password (because of the ssh keys you have set up). Instead, you should immediately see ‘Welcome! This is a restricted Shell Account. You can only copy files to/from here.’ If you see this, then everything is set up correctly.
1-2 weeks before release:
At least one week before releasing, we enter a review period, during which the only
changes made to the code to be released should be to fix errors, rather than to add new
features. A release branch will be created by the release technician, to which ONLY
release-blocking bug fixes and corrections to typographical errors will be pushed. The
release technician should announce a temporary hold on pushes to the dev branch on the
Council list, then create the branch and push it to GitHub using the following commands:
git checkout dev (make sure you start from the dev branch)
git checkout -b release-X.X.X
git push -u origin release-X.X.X
After the release branch has been pushed, the release technician should inform the Council list and ask the maintainers of the TEI Jenkins systems (currently Martin Holmes, Peter Stadler, and James Cummings) to add a build of the release branch so that commits pushed there are properly tested.
Pushes to the release branch should be merged back into dev regularly:
git checkout dev
git merge release-X.X.X
On release day:
Ensure that P5/ReleaseNotes/readme-X.X.X.xml has been
Normally, this will have been created by the TEI Council chair at the point when the repository moved from its "alpha" stage to "beta", so you should not have to do this, but in case you do:
- Confirm the version number for the new release in consultation with Council. TEI version numbers are based on the system used by the Unicode Consortium but with the first digit for major changes, the second for schema-changing revisions, and the third for non-schema-changing revisions. When the first or second digit is incremented, the following digit or digits is set to zero. During initial development, the version number is followed by "alpha"; during the pre-release checking stage, it's followed by "beta"; and when the release takes place, "beta" is removed and the version number has only digits and periods.
- Create the new file by cloning a previous one.
Consult the git log log to check for all the significant changes since the last
release. You can do this by opening a terminal in the root of a working copy of the
TEI repository and running:
git log --after=2015-08-01
where the date is that of the previous release.
- Add the new file into the repository with git add.
Edit the P5/VERSION file to the correct number
This file consists only of the bare version number, followed by "alpha" or "beta":
For the release process, you need to remove the letters from the end, leaving a pure version number:
This changes the release from beta (or possibly alpha) to the actual release version number. After the release process has been completed, the number should be incremented appropriately, and "alpha" added to the end of it:
signifying that the versions built subsequent to the release are now in the alpha stage.
- Announce a temporary freeze on pushes to the release branch on the TEI Technical Council mailing list
In the P5 directory, do
to generate an updated ChangeLog file.
Merge the release branch into master
git checkout master
git merge --no-ff -m "YOUR COMMIT MESSAGE" release-X.X.X
Push your changes to the git repository, git push origin
and watch Jenkins build P5 for you.
This should be the final push for this version, and it will trigger the Jenkins servers into rebuilding the TEI packages. This is where you'll find the Jenkins servers:
Ensure all changes have been committed, built, and successfully
passed tests on the continual integration server
When all builds have completed on both servers, click on the job number of the last build for each of the three TEI jobs to make sure that it was triggered by the commit that you made in the previous step (you should see your own commit message on the build page). Make sure that all builds were successful (they should have green balls next to them).
NOTE: INSTRUCTIONS FROM HERE ON ASSUME THAT WE ARE RELEASING
PRODUCTS ON THE SOURCEFORGE WEBSITE.
Log into TEI server and run the tei-install.sh script:
cd ~/private/git/P5 then run git pull
then cd Utilities (this is where the scripts are)
(running git pull will update the local copy of the git repository because the scripts in that the release technician is using are in git. You should not be editing these scripts locally on tei-c.org instead, edit them and commit them to the git repository. Wherever possible there should be no special local magic files.)
Do the following three steps:
- Install locally: ./tei-install.sh --package=TEIP5-master --version=X.X.X --sfuser=username --install and then go test the version this puts in the Vault.
- If that looks good and everyone agrees then: ./tei-install.sh --package=TEIP5-master --version=X.X.X --sfuser=username --makecurrent and then test that it appears on website correctly.
- If the website looks right then: ./tei-install.sh --package=TEIP5 --version=X.X.X --sfuser=username --upload and then move on to the next step.
In each of these steps, replace the Xs with your release version. Supply your SourceForge user name, and type your SourceForge password when prompted. By default, the script will pull the release package from the first Jenkins server, but you can supply the URL of the other server if necessary; read the script to see the details. Make sure the script completes successfully each time changing the final parameter from --install, to --makecurrent, and then --upload.
Still in ~/private/git/P5/Utilities, run
This will update the eXist-db used by Roma. The script takes no parameters but expects a configuration file in the user’s home directory at ~/etc/tei-exist.conf with the appropriate paths and credentials. For the main server at tei-c.org this should already be set and no adjustments should be needed. This is the database used by Roma, so this operation will ensure that Roma is using the new version. Ask Martin, Peter, James or Sebastian to check the version of P5 used by Roma, to ensure that this has succeeded, and fix it manually if it has not. This procedure is undergoing revision right now, so it would not be surprising if there were a minor glitch here. Eventually, a new process will be put in place and this instructions will be updated.
Check the TEI website and all downloadable files are displaying the
Everything should now be done, so go to the newly released version on the TEI site and browse the Guidelines. Check that your version number is displayed in the footer of the page, and check that at least one change made since the last release is being reflected online. Visit Roma at http://www.tei-c.org/Roma/ , and generate a simple schema to confirm it works, and check that the version number is right in the footer.
NOTE: AGAIN THIS ASSUMES THAT WE ARE STILL RELEASING PRODUCTS ON
Make your release the default downloadable version from
Go to the SourceForge site:
log in, and click the information button on your new release. Make it the default download for all operating systems. Now make sure that the main Download button links to your package.
Update tags on GitHub
Every time a new release is made, a "tag" is created consisting of a pointer to the state of the P5 tree at release time. You can do this from the command line on your own computer. This is how to do it:
git tag -a P5_Release_X.X.X -m "Release X.X.X of the TEI Guidelines."
where X.X.X is your new release. Then
git push origin P5_Release_X.X.X
Make a Release on GitHub
Go to the TEI Tags page on GitHub. You should see the tag you just pushed there. Click on "Add release notes". Give it a title, generally "Release X.X.X" and add a link to the release notes README, which should be at http://www.tei-c.org/release/doc/tei-p5-doc/readme-X.X.X.html. Add a copy of the zipped release from Jenkins. NOTE: THIS STEP WILL BE AUTOMATED SOON.
Inform the Debian Package Maintainer of the new release
Note: This step may change as we review Debian Package Creation The Debian repository can only be updated by its maintainers, so let the Debian Package Maintainer know that your release is done, so they can grab the new packages and add them to the repository.
Update the list of Previous releases of
If you have editing privileges on the OpenCMS system on tei-c.org, add the new release to the top of the release table. If not, ask one of the other Council members who does (currently Hugh Cayless and James Cummings) to make the change.
Update the oXygen-ready distribution of TEI.
This involves building the new package of oxygen-tei, and then updating the distribution file on the TEI server so that the oXygen software knows about the new release. You may prefer to hand this off to one of the maintainers (currently James Cummings, Syd Bauman or Martin Holmes) to do this for you if you're not familiar with the project.
- Check that you have ant (at least version 1.8) installed on your machine.
- Check that the latest versions of the TEI Stylesheets and TEI P5 packages are available for download from the SourceForge Files section, since the oxygen-tei update/upload script retrieves them from there.
- Check out or update a local copy of the source code from https://github.com/TEIC/oxygen-tei to your local system.
- Make sure the build script knows where to find your oxygen.jar file (from the Oxygen installed on your system) by copying the file into oxygen-tei/lib or symlinking it.
- cd into the oxygen-tei folder (it should contain folders called "frameworks" and "jenkins").
Run the ant build process with the "release" parameter:
This builds the plugin using the latest stable versions of P5 and the Stylesheets, then offers to upload the result to the TEI's SourceForge repo to become a release of the TEI-maintained version of the plugin. This also creates an updated updateSite.oxygen file, by retrieving the latest updateSite.oxygen file from the tei-c.org site, and asks the user to provide the new version number before creating a new version of updateSite.oxygen. It also offers to upload that file to tei-c.org, if the user has shell access.
Inform the TEI Technical Council Chair so they can announce the
Once you are sure that everything is working correctly, inform the Council Chair. They will announce the release to the TEI-L mailing list, including the text of P5/ReleaseNotes/readme-X.X.X.xml in plain text form (which can be generated using the "readme" profile for teitotxt), and place an announcement on the Text Encoding Initiative Newsfeed blog in the category of 'News'. They will also update the table of previous releases at http://www.tei-c.org/Guidelines/P5/index.xml#previous.
Lift the freeze on committing changes to the repository
Write to the TEI Council list and let them know that they can once again start committing changes to the repository.
Inform the Stylesheets maintainer that a new release has been
The TEI Stylesheets project depends on the current release of P5, and when a new release appears, the tests which are part of the stylesheet build process will probably fail, breaking the build. The maintainer will have to change the tests in accordance with the new release of P5. This can only be done after the release process has been completed. Refer to the TEI Council chair to find out who the current Stylesheets maintainer is.
Increment the build number for the next release cycle
After the release process has been completed, the release number in the repository needs to be updated. Edit the P5/VERSION file to the correct number. This file contains nothing except the bare version number. It should be incremented appropriately, and "a" added to the end of it, so if for example the release was number 2.8.2, you would change the number in the file to:
signifying that the versions built subsequent to the release are now in the alpha stage.