Windows 10 Preview on the Spectre x360 - Fixing missing devices

This year's freebie at Microsoft Build 2015 conference is the HP Spectre x360 laptop, which came with Windows 8.1 installed. The idea is to take it away and update it to Windows 10 to allow all us developers to get to grips with the new features of Windows 10, VS2015, etc.


Like many other geeks I've never been particularly happy with the sort of crap you get pre-installed on laptops. Even when you don't get true bloatware, there are usually things like replacement Wifi management apps or manufacturer personal data collection support apps. Also, I've always preferred to do a clean install of a new version of Windows rather than an upgrade.

So, that meant starting from scratch, wiping the windows 8.1 partition and installing Windows 10 straight from a USB stick.  Install went fine, but there were a lot of unrecognized devices in device manager once it had finished.

What's missing? Where are the downloads?

Intel SMBus Controller and other Intel devices

https://downloadcenter.intel.com/product/1145/Intel-Chipset-Software-Installation-Utility

HP Wireless Button Driver

ftp://ftp.hp.com/pub/softpaq/sp58501-59000/sp58720.exe 

Audio Driver

http://ftp.hp.com/pub/softpaq/sp70501-71000/sp70653.exe

Intel Virtual Buttons Driver

http://ftp.hp.com/pub/softpaq/sp70001-70500/sp70349.exe 


This might not be entirely complete, as I have put it together from memory and the things I have left in my download folder after performing the process. I don't feel like going back and installing agian just now to check :)

There was one other driver missing, but that got picked up by asking it to update and automatically find drivers for my device.

You can find other downloads for the machine here

http://support.hp.com/us-en/drivers/selfservice/HP-Spectre-x360-Convertible-PC-Series/7527520/model/7791778#Z7_3054ICK0KGTE30AQO5O3KA30R1

Building Pdf2HtmlEx for Windows

Pdf2HtmlEx (https://github.com/coolwanglu/pdf2htmlEX) is a great open source project for accurately rendering PDF files into modern, compact HTML. Getting a Windows build of what is really a Linux orientated open source project is a bit of a challenge however.

If you fancy just downloading a binary, then perhaps head over to this site, where you will find some builds. For me, that wasn't going to be sufficient - if this is going to go into a production environment then I need to be able to build a new version if and when we need to fix bugs or update to the latest build. I wanted to build from source.

Fortunately Marc Sanfaçon has done most of the hard work in working out what other packages have to be built and how to build them under mingw. I set out to follow his instructions and along the way I fixed any gremlins I came across and turned them into a bash script that will perform the whole build unattended from start to finish.

Before you start, make sure you have installed :

• Mingw from here. Leave the defaults (like installing to c:\mingw) alone in the first page of the installer, then select at least 'mingw-developer-tools', 'mingw32-base',  'mingw32-gcc-g++' and 'msys-base' in the package selection.
• Python from here. Actually, the final built product doesn't depend on Python, but building libfontforge seems to require it. In the end it was easier just to install Python than spend more time hacking at the build system trying to remove the dependency. You also need to add Python to your PATH.

Now, start an MSYS shell (there's a batch file in c:\MinGW\msys\1.0) and you will have a bash shell in your home directory (C:\MinGw\msys\1.0\home\<username>).

Copy the script below into a .sh file and run it. You'll need an active internet connection for source to be downloaded, but other than that all you need do is wait. Once the process is complete, the Pdf2HtmlEx.exe file (and the dll files and data files it depends on) will be in the 'pdf2htmlEx.Install' subdirectory of your home directory.

I would really have liked to make a statically linked version of the executable, but it didn't seem a simple change to make, so that will have to wait for now (or for someone with a deeper knowledge of CMake, configure, etc).

Anyway, the script is here.

Moving source control from svn to git

Ok, git tricked me here by having a clone operation that isn't really a clone. In my book the result of a clone should be identical to the original. In the git world that's not quite true - the clone doesn't have branches like the original. Sure it has the branches tucked away somewhere inside the .git folder, and you can check them out from the remote and track them as a local branch if you need to, but if you list the branches available to you you'll find just master. Now, if you take it one step further and clone the first clone you made, the second clone seems to have no visibility of the branches at all...

Now, in general use, the no-branches-in-a-clone workflow makes a lot of sense - you don't want to know about someone elses branches by default, it would just get too messy. However when you want to clone a repository to move it to a new server (or in my case to move it from svn to git), the git clone operation is a problem.

Fortunately the solution is to run a fairly simple script to checkout each remote branch and track it as a local branch.

for remote in `git branch -r | grep -v trunk `; do git checkout -b $remote $remote ; done




I'm working in a repository that came from svn - hence the 'trunk' check. If you are pulling from git, replace the grep -v trunk with grep -v master.



To do the actual pull from svn to git is pretty simple using the git svn command


git svn clone "svn://server/repository"  "repository" -T trunk -b branches [-t tags]




Once you've done that, do the remote branch checkout script above to track all the remote branches. Note that it's best to access your svn repository via a network protocol (svn:// https:// svn+ssh://) rather than a raw svn repository on a local path - this is because git svn only understands really old svn repository formats, so will fail with repositories made by a recent version of svn.


I'm using gitolite to host my git repositiories on my linux box - that seems to expect raw or 'bare' git repositories. I used the standard gitolite methods to add a new repository, then deleted the newrepo.git folder that got created and copied the .git folder into its place. Fix the ownership/rights on the new folder if necessary (chown -R git.git newrepo.git ; chwon -R 700 newrepo.git) and you're done.

Rewriting History - in an SVN Repository

I've been thinking about moving one of my personal projects from SVN source control to git. Mainly because of the offline commit ability in git which has been 'coming soon' for ever in SVN.

Git has a fairly nice import functionality to pull an SVN repository into git, but it's really designed around the standard SVN repository layout of /trunk, /branches and /tags. My repository doesn't look like that - for some reason (most likely it was the first time I had ever used SVN, many years ago) I created it without any of those conventional folders. Bad idea #1....

My SVN repo also has a lot of separate projects in it as folders at the top level - this is probably bad idea #2. Then, a couple of years back I needed to branch one of my projects, so I created a top level /trunk folder, moved all the folders into that, created a top level /branches and made a branch there. Bad idea #3 I'm thinking.

So, my repository is a mess and the project I'm most interested in - SharpCap - can be found in /trunk/SharpCap and /branches/SharpCap/[branch]. Yuck.

Maybe I could pull the whole lot into git and prune and chop it into shape afterwards - however my git-fu is still weak and I'm not going to try that yet. Instead I decided to try to rewrite the history of my SVN repo to make it fit the standard layout more nicely. Of course I'm doing this on a *copy* of the SVN repository, not the real thing.

Firstly, I'm working on Linux - Ubuntu 12.04 LTS to be precise. I expect all the steps below can be run on Windows, but you'll be messing about installing perl and python and goodness knows what else - easier on Linux by far.

Anyway, first the tools for the job :

• svnadmin - you'll already have this on your system, if not: apt-get install svn
• svn-dump-reloc - http://search.cpan.org/~salva/SVN-DumpReloc-0.02/bin/svn-dump-reloc
• svndumpfilter2 - http://svn.tartarus.org/sgt/svn-tools/svndumpfilter2?view=markup
• svndumpfilter3 - https://github.com/bmwiedemann/svndumpfilter3/blob/master/svndumpfilter3

The svndumpfilter tools you'll just need to download the script files - svn-dump-reloc can be installed via cpan - install perl via apt-get if you don't have it, then run cpan, work through the initial questions and then do install SALVA/SVN-DumpReloc-0.02.tar.gz.





The basic technique is to use svnadmin dump to dump the repository to file, use one of the filter tools to modify the repository, then use svnadmin load to bring the modified dump back into a new repository. So, assuming that you repository is at /home/svn/myrepo, you might do this...

svnadmin dump /home/svn/myrepo > /tmp/myrepo.dmp
cat /tmp/myrepo.dmp | svndumpfilter3 --untangle /home/svn/myrepo /path/to/keep/in/repository > /tmp/filtered.dmp
mkdir /tmp/filtered && svnadmin create /tmp/filtered
svnadmin load /tmp/filtered < /tmp/filtered.dmp

What's going on there? First we dump the original repository to a file, then we pass it through svndumpfilter3 to only keep a particular path or paths, then load back into a new repository. svndumpfilter3 needs to know about the location of the actual repository the dump file came from - in some cases it goes back to the repository to dig out extra information to help it deal with moves, copies, etc in the repository.

So, in my case the svndumpfilter command is

cat /tmp/myrepo.dmp | svndumpfilter3 --untangle /home/svn/myrepo /SharpCap /trunk/SharpCap /branches/SharpCap > /tmp/filtered.dmp

This pulls out the bits of the repository I'm interested in and throws out the rest. 

Now, it's not quite as simple as it looks to reload this dump into a repository - if you try it, you'll find it just fails. This is because we haven't included the creation of /trunk or /branches in our filter, so the first revision that tries to do something into one of those folders will fail to load because the folder is missing. You'll get an error like this :

svnadmin: File not found: transaction '307-8j', path 'branches/SharpCap'

Here's how to step around that by creating the parent folders first.

mkdir /tmp/filtered && svnadmin create /tmp/filtered
svn mkdir -m "make trunk" file:///tmp/filtered/trunk
svn mkdir -m "make branches" file:///tmp/filtered/branches
svnadmin load /tmp/filtered < /tmp/filtered.dmp


Now, while svndumpfilter3 seems to be the best choice for pruning the repository (it gets confused much less often than svndumpfilter2 or the original svndumpfilter), it doesn't have an option to drop empty revisions from the dump file. If you've pruned out a significant chunk of a repository, you'll most likely want to get rid of those, and this is where svndumpfilter2 comes in handy.

cat /tmp/filtered.dmp | svndumpfilter2  --drop-empty-revs --renumber-revs /tmp/filtered trunk branches SharpCap > /tmp/renumbered.dmp
So, what we've done there is reload the 'filtered' dump into a temporary repository - this is because we need the repository for svndumpfilter2 to work with - and then process the dump again to drop the empty revisions. By specifiying 'trunk' and 'branches' and 'SharpCap' to svndumpfilter2 I have told it to include everything in the source dump (add tags too, if you use those), so I'm just using it to renumber revisions rather than filter anything here.


 With me so far? Good... Now for the tricky bit - we need to re-arrange the history of the repository folder structure. Basically the mapping I want to do is as follows:

/SharpCap -> /trunk
/branches/SharpCap/<branch> -> /branches/branch

First try is just to use svn-dump-reloc three times - ie.

cat /tmp/renumbered.dmp | svn-dump-reloc '/trunk/SharpCap' '/trunk' | svn-dump-reloc '/SharpCap' '/trunk' | svn-dump-reloc '/branches/SharpCap' '/branches' > moved.dmp

This should move any item that was historically in /trunk/SharpCap into /trunk, and the same with anything in /SharpCap. The final bit should move anything in /branches/SharpCap into /branches. Unfortunately the dump won't load. 

The reason the dump won't load is that I have a very interesting revision in it right now. Before the relocation, that revision used to say 'copy the contents of /SharpCap to /trunk/SharpCap and then delete /SharpCap'.  After the relocation it now says 'copy the contents of /trunk to /trunk and then delete /trunk'. Ooops. In the actual dumpfile, the revision looked like this :

Revision-number: 149
Prop-content-length: 126
Content-length: 126

K 8
svn:date
V 27
2010-09-14T19:39:50.182787Z
K 7
svn:log
V 26
move main folders to trunk
K 10
svn:author
V 5
robin
PROPS-END

Node-path: trunk
Node-kind: dir
Node-action: add
Node-copyfrom-rev: 148
Node-copyfrom-path: trunk

Node-path: trunk
Node-action: delete

So, of course the next revision after that one that tries to do anything to /trunk fails with the old

svnadmin: File not found: transaction '159-4f', path '/trunk'

 error. Basically all I needed to do was to get rid of this revision from my dump file - vi was quite sufficient to do that on my (~150Mb) dump. For bigger dumps you might need to use a more powerful editor or work out another way to remove it - the svnadmin dump command allows you to specify a revision range, so you could load the renumbed.dmp file and then dump it in two parts -r 0:148 and -r 150:HEAD sort of thing, then cat the two together.

So, finally I have an svn repository that has 'always' had the structure I want. Now to try loading it into git...

Fixing UK keyboard mapping for Xrdp

Xrdp is a pretty nice tool for accessing a Linux box from Windows - you can use the Windows built in remote desktop client rather than having to install some flavour of VNC. It all just works - nearly...

The trouble is that Xrdp doesn't seem to support X windows keyboard mappings properly. I pretty quickly noticed that my ",@,#,~,\,|,£ characters were behaving as though I had a US keyboard not a UK one.

First point of call was the X keyboard settings UI, which oddly was blank with no additional layouts to choose from or options to set. Going to the same settings under an X login on the machine itself was fine. Hmmm...

A little digging revealed this post which outlines the approach to take to generate a new keyboard map for xrdp. For en-gb you need to create a 'km-0809.ini' file and install it into /etc/xrdp. Unfortunately, while this keyboard map fixed the #,@,| etc characters, it broke the arrow keys, page up, page down, home, end and possibly a whole load of other utility keys too.

Fortunately text based configuration files and a diff viewer (Meld) came to the rescue. I was able to cherry pick the changes that I did want from the 0809 ini file and apply them to a copy of the default US english (0409) ini file. I found the list of key codes that I wanted to move across by using the 'xev' tool to record keypress information on the problem keys.

Once the file was finished, it only needs to be copied to /etc/xrdp/km-0809.ini and given the right ownership and permissions. Then restart the xrdp service and you should be done

Here are the commands (on Ubuntu 12.04) to setup the permissions etc and restart the service

sudo chown xrdp.xrdp /etc/xrdp/km-0809.ini
sudo chmod 644 /etc/xrdp/km-0809.ini
sudo service xrdp restart

Here is the corrected UK English keyboard file : km-0809.ini

Please note that comments on this post have now been disabled due to a big increase in the number of spam comments recently (Jan 2019)