Skip to content

Next Release 5.6.0 #600

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
geoffmcl opened this issue Sep 17, 2017 · 23 comments
Closed

Next Release 5.6.0 #600

geoffmcl opened this issue Sep 17, 2017 · 23 comments

Comments

@geoffmcl
Copy link
Contributor

@balthisar, although, I had originally arbitrarily set this for Sept 20, 2017, I do not think I would be ready for that...

Will try to do a detailed analysis of the open issues, over the coming days, but at the very least would like to close #597 first... but there are probably others that are important...

So propose moving it out at least a few weeks, if not a month, or months... As usual I could do the release notes, and upload windows and unix binaries when we get there...

What is your schedule like? What would be a good target date for you?

We need you 100% for the macOS... thanks...

@balthisar
Copy link
Member

@geoffmcl, no problems holding off. It would have been nice to have an autumn solstice release, but maybe we can make Canada happy by honoring their Thanksgiving tradition for a release?

It looks like there's some low-hanging fruit that can be cleaned up. I'm not sure how much I can commit to them, but I'll check in and can definitely support macOS binaries, doc updates, gh-pages, etc.

@geoffmcl
Copy link
Contributor Author

@balthisar had to look it up, but went for thanksgiving, Nov 23, 2017, I think ;=))

Will work toward clearing as many issues as possible, as usual...

Also created a 5.7, for about 6 months later - May 24, 2018 - but just a random date... will start to move issue unlikely to make it to release 5.6 to there... thanks...

@balthisar
Copy link
Member

Sounds great, @geoffmcl.

@balthisar
Copy link
Member

We're up to .70! This is going to be our most improved release yet.

@geoffmcl
Copy link
Contributor Author

@balthisar, now up to .85, and have reviewed the new verhist.log, and a test sample 5.6.0.html release notes, and there is certainly some great stuff... some 436 commits by 12 authors... most by you, 306...

Tidy Release Steps

I though I already had this somewhere, but could not now immediately find it... Also note several steps are very order specific...

  1. Change version.txt to 5.6.0, and push the change
  2. Update README/verhist.log and push
  3. Create tag 5.6.0, and push
  4. Create branch release/5.6, and push
  5. Generate 5.6.0.html release notes, in binaries/release_notes, and push
  6. Generate release binaries, and upload to release - (a) Win, (b) Mac, (c) Unix
  7. Update binaries/_data/tidy.yml and push
  8. Merge release/5.6 to master, and push
  9. Update next version.txt to 5.7, and push

I have easy scripts to do 2. and 5., so I could take it up to 6 (a)... advise if I missed anything... bad order... appreciate feedback on this list...

Are there any special issues that you think must be included, if at all possible... which may put the schedule back? I have none...

But if none, then we need to actually choose a day... suggest Sat or Sun, 25 or 26? Which is better for you? Or some other? thanks...

@balthisar
Copy link
Member

@geoffmcl, this list looks complete to me, other than api.html-tidy.org, which I can take care of.

I might take a look at some of the readme's to make sure that they're up to date, first, as they're easy to overlook. That would be prior to the 5.6 tag/push.

As far as order, feel free to merge to master if not all of the binaries are present. I can add the macOS binaries after the fact.

If you want to merge #645 before the push, I'm okay with that, despite wanting to know your vision of the future.

You're working +5 hours or so, so from my perspective, Friday, Saturday, or Sunday are all good, as long as my dear wife is tolerant.

I really think that this is going to be our biggest update yet, and I'm looking forward to it. I wish we had more help, but for two guys separated by an ocean and having vastly different views on some things, I think we manage to do pretty damned good!

And... I hope you're feeling better.

@geoffmcl
Copy link
Contributor Author

@balthisar yes, feeling better... thanks...

Yes, had forgotten the api docs update... glad you will take care of that...

Generally, Fridays are very short computer days for me... shopping takes over...

Will probably skip #645 for now, and move it out to next 5.7...

So let's go for Saturday, with Sunday fall back... as stated I will try to kick off with 1 to 6a...

Agree for two guys separated by an ocean, with different views on some things, we do do OK ;=))

@geoffmcl
Copy link
Contributor Author

@balthisar as expected, little time today, but do want to float two ideas for this release...

  1. Binary Files use a naming convention
  2. Add SHA1 hash for binaries

The first is because I am thinking of at least 4 zips for windows, with names like tidy-<version>-<tool>-<bits>.zip, so I would upload -

  1. tidy-5.6.0-vc14-64b.zip
  2. tidy-5.6.0-vc14-32b.zip
  3. tidy-5.6.0-vc10-64b.zip
  4. tidy-5.6.0-vc10-32b.zip

And also some variation for my Ubuntu builds, 32 and 64 bit...

Maybe even add a MinGW build, or builds, and Raspbian build... and there are possibly other Windows variations, like /MT static instead of /MD... etc

What do you think? suggestions... thanks...

On 2., as you know we have had a virus report, a false positives I think, but remember we did discuss adding at least a SHA1 hash...

For each I could also upload like a tidy-5.6.0-vc14-64b.zip.SHA1 text for each zip at the same time...

Could this either hash value, or at least the download of the corresponding SHA1 file, be added to the binaries site?

I seem to remember we did discuss this somewhere, and you would look into it... any other quick ideas, other than adding the SHA1 file?

Off shopping now... be back soonest...

@balthisar
Copy link
Member

@geoffmcl, Sure, I can look into modifying the binaries to do just that.

One question about all those versions, though: are they all needed? I suppose the vc10's are, because I think I remember reports that some older OS didn't have the required Windows libraries? And I suppose there are still 32-bit Windows, and Linuxes. But considering what the typical user* who's looking for binaries (rather than building for himself) wants, why would he ever care about a MinGW build, or static versus dynamic builds?

In any case, I'll make sure the binaries system works with whatever you throw at it, and you're the one generating the Windows builds, but I just wonder if you're overworking yourself with all those builds!

@balthisar
Copy link
Member

@geoffmcl, I've updated binaries.html-tidy.org to include SSH256 hashes, and I've added hashes for all of the existing binaries. Decided not to use SHA1 as it’s insecure.

geoffmcl added a commit that referenced this issue Nov 25, 2017
geoffmcl added a commit that referenced this issue Nov 25, 2017
@geoffmcl
Copy link
Contributor Author

@balthisar ask for A and get B, sans discussion!, but thanks, I think ;=))

Have now completed 1 to 6.a.1 at least... slight delay while I adjusted my tool chain for the SHA256 generation in Windows...

Tried Power Shell, but that reminded my how much I dislike that interface, and forked github.com/B-Con/crypto-algorithms to build my own...

And that setting up of a tool chain is why now adding multiple builds is not really overwork ;=)) Is FUN even...

And if you read through the appropriate issue comments, I think more choices is better, plus 32-bits, etc... I even still run 2 32-bit linuxes... Will add more bins over the coming days...

Out for a nice cup of coffee, in the sun, but getting quite cool... later... thanks...

@balthisar
Copy link
Member

@geoffmcl, sorry about that. I assumed OpenSSL was installed by default on Windows. It's on my Windows 10 setup, and works in cmd; no Powershell required.

@geoffmcl
Copy link
Contributor Author

@balthisar no problem... as you can see I rather enjoyed finding, researching, building and setting up my Windows tool chain... but takes time...

But that did make me ponder about other Windows users... what are they likely to have available by way of checksum generators, and checkers?

I certainly do not think it is installed by default in Windows 10, at least not in my free upgraded to 10... and in fact did not even find a binary install that included it... but did not look too hard now I have my own ;=))

So to be very sure added the small files tidy-5.6.0-vc14-64b.zip.SHA1 and a tidy-5.6.0-vc14-64b.zip.MD5, that they can download with those checksums, for verification that the zip has not been tampered with...

Or even check all if they are really vexed about security LOL

But am happy that the binaries site only has one displayed... looks more professional now... thanks...

And also seems sha256 is not in my main Ubuntu 14.04 64-bits, and tried the suggested $ sudo apt-get install hashalot, but that does seem the right thing! What am I supposed to be using? I do have md5sum and sha1sum, which seem to work fine... AH HA!, I do have sha256sum, so maybe this is ok...

But in trying the --check <file>.SHA256 maybe what I have presently uploaded is in the wrong format! Seems it should be just checksum file, or something... Will experiment more but out of time tonight...

Maybe you will get a chance to complete some more of the release steps... thanks...

@balthisar
Copy link
Member

@geoffmcl, I'll work on macOS binaries presently. As you mention, the hash files in the current "releases" are tests, and I'm not going to link to them.

On Windows 10 (updated from Windows 7 Pro clean install just a month ago), the program I use is openssl, which was either installed by default, or installed via one of the Windows options. There's nothing else installed on this machine except for Chrome, Notepad ++, and Visual Studio 2017.

openssl is also in my default Ubuntu, and on my default macOS, although I may have been installed as a dependency to something else, because it's ubiquitous and the defacto tool for everything, including sha1. I'm really sorry I didn't think to ask that it would be a problem.

I'll merge to master, too, since you're sleeping now, and update the binaries site.

@balthisar
Copy link
Member

Update: www.html-tidy.org updated, binaries.* updated with your .zip and my .dmg, sha256 added for both, api.* updated for 5.6.0 and next, next merged to master, next bumped to 5.7.0, and I think we're good.

I didn't generate RPM's or binaries for other versions of Windows simply because I don't use them. It's good that they're there until the distros catch up, though. If you have time for a quick lesson, I can take the load off next time.

@geoffmcl
Copy link
Contributor Author

@balthisar did not get very much done on tidy today, but note you have been very busy ;=))

As mentioned, the binaries site looks great, and yes did not particularly intend you link with the --check file checksum files, although... more on that below...

But wow, suddenly realized you are talking about an openssl.exe! And yes that is installed, in my case as part of the Subversion install, a long time ago, which is around the date of this system's original creation -

 Directory of C:\Program Files (x86)\Subversion\bin
2012-02-16  09:24           381,006 openssl.exe

So, yes, I now do have an alternative way to generate a sha256 checksum, for a zip, say -

F:\Projects\install\msvc140-64>openssl dgst -sha256 tidy-5.6.0-vc14-64b.zip
WARNING: can't open config file: /usr/local/ssl/openssl.cnf
SHA256(tidy-5.6.0-vc14-64b.zip)= 59ac22427a3d759c72108814f6a8f0e98511b12f87744ca19e0a500a00687be8

But the <file>.SHA256 I have uploaded has quite an interesting purpose, now I have the correct the format, for this check file, and that is you can just download the <file> and <file>.SHA256 to some common place, and running the following should give an OK, like -

$ sha256sum --check <file>.SHA256
<file>: OK

That is, it gets the checksum and actual file name from the check file, and does the generation and verification... Now maybe openssl.exe has a similar functionality, not yet found, but my own sha256_test.exe does...

But as stated, I am also happy the just have the sha256 checksum visible on the binaries site, but will continue to add the check files to the github release site, for this sort of dual download test process...

Although, if the binaries site also offered them, the check files, as a download, it would be one less pieces to have to add to tidy.yml each time if we adopted the <file>.SHA256 convention for all... but I am ok with either...

What do you think?

And thanks for doing the api site... this looks great too... and the master merge, and next to 5.7.0...

As usual off to dinner - sleep is much, much later...

I too think we are good ;=))

@balthisar
Copy link
Member

@geoffmcl, is there some checksum verification process that's automatic that I don't know about? I'm a bit slower as I age... I think I understand that you can download two files -- and .sha256, run a simple command, and get an okay. Then people don't have to verify character by character the actual checksum? Which is great, considering I only look at the first few and last few characters when doing it manually!

Is this a common practice that I'm just not aware of, or is this just a thing that only one or two people know about? Either way, there's no work; we can upload anything we want to github and the binaries page; I'm just curious if this is a workflow I'm ignorant of.

@geoffmcl
Copy link
Contributor Author

@balthisar well I too was ignorant of it too until a few days ago, when you pushed me into finding and using sha256sum... and through the --help command got curious about the --check option...

But it is really picky about the format of the file, and I still have to fix what is there... it will not accept windows CR/LF... will fix that...

So after downloading the 2 files -

~/downloads$ dir tidy-5.6.0*
-rw-rw-r-- 1 geoff geoff 1174012 Nov 27 15:24 tidy-5.6.0-vc14-64b.zip
-rw-rw-r-- 1 geoff geoff      90 Nov 27 15:25 tidy-5.6.0-vc14-64b.zip.SHA256
~/downloads$ sha256sum --check tidy-5.6.0-vc14-64b.zip.SHA256
: No such file or directory64b.zip
: FAILED open or readip
sha256sum: WARNING: 1 listed file could not be read
~/downloads$ dos2unix tidy-5.6.0-vc14-64b.zip.SHA256
Using 'fromdos tidy-5.6.0-vc14-64b.zip.SHA256'
~/downloads$ sha256sum --check tidy-5.6.0-vc14-64b.zip.SHA256
tidy-5.6.0-vc14-64b.zip: OK

Look how angy it gets, with very misleading error messages, over line endings...

But after I ran dos2unix, was as happy as pie...

Now I do not know how many people know about this, but it seems an easy way to verify a checksum without actually ever seeing the checksum... and just tested changing just one letter in the .SHA256 file and got -

tidy-5.6.0-vc14-64b.zip: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match

Although I did not know about this --check option way back when I wrote a windows md5 app, I did include an option to do the compare for me - it was just too much trouble, and fraught with possible mistakes, to do a physical visual compare -

C:\Users\user>md5
md5: Compile on Nov 28 2008, at 16:59:18 - Usage: [Options] file
  --help     -h (or -?)  # this brief help, and exit.
  --test     -t          # run the self-test (A.5 of RFC 1321), and exit.
  --t-values -l          # print the T values for the library, and exit
  --version  -v          # print the version of the package, and exit.
  --digest=  -d=<digest> # compare file digest with this value.
  --html                 # ouput information in HTML table form.
  filename               # output the MD5 sum for the file.

Just noted, nearly the 9th aniversary of that app...

And certainly note quite a lot of, usually ftp type, download sites do include these check files... so they too offer the possibility... and it works for md5sum and sha1sum the same way...

Anyway, as stated, I am also happy with just showing the SHA256 on the binaries site...

But will continue to add the check files for each on github... and will try to get around to doing it also for all the previous files... unless you beat me to it ;=))

@geoffmcl
Copy link
Contributor Author

Tricky Geoff ;=))

Added an option to my sha256_test app to directly output an out file, -o out.SHA256, rather than using redirection as before, and it has no line endings!... Who can dislike that?

Tried that out with linux sha256sum, and it is happy ;=))

Will now do the same for my new md5_test and sha1_test apps... will than be able to add the check files for all, hopefully suitable for all OSes...

Hope you get a chance to check in the macOS... thanks...

@geoffmcl
Copy link
Contributor Author

@balthisar have now added the 4 windows zips I promised, including the appropriate check files...

Found a tiny tiny problem with the MSVC10 build, and had to apply the following patch -

diff --git a/src/tidylib.c b/src/tidylib.c
index 85d59e2..ce72acd 100644
--- a/src/tidylib.c
+++ b/src/tidylib.c
@@ -1436,8 +1436,8 @@ int         TY_(DocParseStream)( TidyDocImpl* doc, StreamIn* in )
     Bool xmlIn = cfgBool( doc, TidyXmlTags );
     TidyConfigChangeCallback callback = doc->pConfigChangeCallback;

-    doc->pConfigChangeCallback = NULL;
     int bomEnc;
+    doc->pConfigChangeCallback = NULL;

     assert( doc != NULL && in != NULL );
     assert( doc->docIn == NULL );

Have not pushed it to master nor release/5.6.0... will get around to fixing it in next... but will face the problem should a user try to compile the release with MSVC10... likelihood seems very low...

Have now generated all the check files, for all the binaries, and could upload them, but am waiting for your choice on whether to support this in the binaries downloads... that is changing the SHA256 from a text string to a link...

And if possible, instead of the long filename being the link text it just be something like -

Name	Last Modified	Size	Description	SHA256
replace with
Name	Last Modified	Size	Description	Checksums

Where Checksums could just be 3 links, like -

... SHA256 SHA1 MD5

Or is that too busy, or too much trouble... look forward to your ideas... thanks...

@geoffmcl
Copy link
Contributor Author

@balthisar have now added more check files for releases, but only for the Windows binaries at this time, since this is where people perceive the biggest threat from viruses... rightly or wrongly...

And made mention of the reason for these text files in the comment...

But, as indicated would add them for all binaries, if you agree... please advise... thanks...

Also suggest we now delete intermediate releases, namely 5.1.4, 5.1.25, 5.1.24, 5.1.8, from github, and from binaries where appropriate... What do you think? Thanks...

@balthisar
Copy link
Member

Sorry I've been unresponsive, @geoffmcl; I've been out of town learning my competitors' secrets (actually it was an open conference). I can't promise to respond much tonight, maybe by the weekend...

Thanks for the MSVC 10 check. I'd thought I'd taught Clang to find all of those for me, and of course I build release and test on Windows, but, sigh, with MSVC 17!

What's the normal workflow for people to check hashes? I've never checked with anything online. In the past, and sometimes today when things aren't signed, I'd just visually compare the hash. I never realized there was a workflow for checking a resource online. My instinct would be not to trust a specialized tool to check online (Trojan horse), but if it's built into, say, openssl, then trust would be high. These days almost everything is signed, and I usually check with spctl (which is probably Mac only).

The macOS installer I build is signed with an official Apple developer ID, for example. I suppose I could sign the .dmg package container, but I'd not thought of that. The hash works for now, though.

We could consider signing as an alternative to hashes (future thinking, not right now), but this gives us issues with archiving private keys and the like. Actually we kind of have similar issues with our domains, which are renewing shortly. If I get whacked by a robot really good, I'd hate to see the domains lost because of the lack of Jim.

I'm not sure why we kept those intermediate releases. There was a compelling reason at the time; but I'd agree to removing them from both binaries.* and releases. I think we have to delete the tags. I'll look into it by the weekend at the latest, unless you beat me to it.

@geoffmcl geoffmcl removed this from the 5.5 milestone Dec 5, 2017
@geoffmcl geoffmcl added this to the Indefinite future milestone Dec 5, 2017
@geoffmcl
Copy link
Contributor Author

geoffmcl commented Dec 5, 2017

Just moved the milestone, but for no particular reason chose Indefinite...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants