Category Archives: Uncategorized

Using Twitter API with Python

My brain never rests. One of the ways I relax is finding solutions to problems. Sometimes thats with crosswords or old episodes of Poirot on ITV3, sometimes thats with writing code. I particularly like using Python.

I use Twitter all the time, for finding out all the latest news and political gossip in astronomy. So, I thought I’d have a bash at using Twitter through python. You used to be able to access Twitter just by using your account username and password, with their basic auth in Twitter API v.1.0. Not any longer. Version 1.0 is now defunct and to do interesting things with Twitter API you have to use version 1.1 which requires OAuth.

This can be a whole world of pain, so I’m writing my findings here to help anyone experiencing similar discomfort.

1. Find a Twitter library

Various people have written libraries to access twitter (I tried writing my own and it was a thankless task). I would recommend using the simply named twitter package. You’ll also need a python way of logging in via OAuth, so I would recommend using OAuth2.

They are both on PyPi so you should be able use:

2. Get developer credentials

You’ll need a Twitter account to go much further with this tutorial. Once you have that:

  1. Log into the Twitter developer site,
  2. Create a new app by clicking on your icon in the top left
  3. Click “My applications” in the menu,
  4. Click the button “Create a new application”
  5. Fill out all the details, sign your life away and click “Create you Twitter application”

You should now be redirected to a page with all the information about your new app, including “OAuth settings”. This page has the 4 horsemen of the OAuth apocalypse:

  • Consumer Key,
  • Consumer secret,
  • Access token (often called “OAuth token”)
  • Access secret (or “OAuth token secret”) – these last 2 in the lower section

3. Put it together

You are now ready to have a play with Twitter from python. Open up a python shell:

Substitute the 4 horsemen into those capitalised variables and hopefully that won’t give you any trouble.

Now you can take Twitter for a spin. There are loads of interesting things you can do listed on the Twitter REST API v1.1 help pages.

I really wanted to have a look at various #hashtags without being rate limited. The way you do this is by using the Twitter Stream and not the Twitter Search. This lets you enter hashtags, words or phrases and then sit on the stream of tweets happening which contain the words you are tracking.

I’ve popped my 4 horsemen into a dictionary called config.

Once you open a stream, tracking certain words, it will carry on forever. For this reason I stop the for loop which checks the stream with a timeout, in the variable stop (a datetime object).

Tip: You can’t look backwards in time in a Twitter stream, only forwards in real-time. If you want to look back in the Twitter feeds you’ll have to use search.

Thats my recent findings. There are so many more things you can do with the Twitter API, but that is just a taste.

A fool on a fool’s errand: Why STEM activity is vital for the future of society

A recent Nature article has slated STEM (Science Technology Engineering and Maths) activities to engage with young people, calling it a “Fool’s errand”. It’s author, Colin Macilwain, was an engineer but for the past 17 years has spent his life working for the dark-side (i.e. Nature and science policy think tanks).

“What no one asked was whether these many activities actually benefit science and engineering, or society as a whole. My answer to both questions is an emphatic ‘no’.” states Macilwain. He goes on to say “Start by asking why no such government-backed programmes exist to pull children into being lawyers or accountants. The obvious answer is that there is no need: young people can see the prospects in these fields for themselves.”

I was considering writing a blog post gently explaining why I disagree.

But not today. This man is wrong and his views are ill-informed.

I am involved in science communication and public engagement in UK. A portion of my job is designated to create STEM activities using the LCOGT telescopes.

However, I am an exception in UK. Most of my colleagues do public engagement because they enjoy it and see value in it, not because they are paid to do it (they aren’t). There is very little funding in UK for these sorts of activities. In the US there is considerably more leading to considerably more “education and outreach specialists”.

In astronomy it is certainly not the case that the goal of STEM activities is to produce more astronomers. That would be totally ludicrous. There are currently ~600 professional astronomers in UK. The public engagement activities I do annually reach more children and young people than that and I certainly do not tell them to all become astronomers.

Here is the crux of my case against the article by Colin Macilwain: we are trying to increase the scientific literacy of the whole population, not to push people into scientific careers.

1. Science is everywhere

I think science is amazing. Unlike history, law or language, science would continue to do its thing without humans. It is the fabric of the natural Universe we are part of. Can you ignore that when you let go of something it drops to the floor, that flowers follow the Sun, or that we only ever see one side of the Moon?

Macilwain suggests that students never need encouragement to follow careers in law and accountancy unlike science, therefore we should not encourage them into science because there are no jobs. I wonder if his argument  applies to apprenticeships too?

The occupations which attract the highest salaries are artificially inflated because of our reliance on them in this material world. 5 years ago I am sure Macilwain would have considered banking to be one of the top professions, along with stock brokering.

Many students are attracted to courses in law and accountancy because they can make money and there is a clear career path.

When you tell someone about a scientific idea, you change their view of the world forever.

Does the same apply to the description of VAT or conveyancing?

2. We have monkey brains

Not the chilled monkey brains from Indiana Jones. The human brain is a big problem solving engine. It is excellent at finding patterns and wants to work out why what we see around us happens. When my little sister was a toddler her favourite word was “Why?”. No answer was sufficient and would always be followed by “Yes, but why?”.

School curricula in US and UK is about learning facts, largely because it is easy to test facts. The UK education minister Michael Gove MP, wants the UK to have 100% of student marks to come from exams because he values learning over everything else. Gove has no time for coursework which he believes makes it easy for students’ to attain higher grants. What he is forgetting is that an exam largely tests recall and rarely tests understanding, whereas a presentation can test both.

Gove and Macilwain are similar in this respect. They appear not to value people forming their own questions about what they have learned.

When we do STEM activities in schools and with the public they are designed to make the audience question what they do and what results they get. In this way you can build understanding which is a far more useful commodity than storing facts. Obviously to be a scientist you need both, but the thing I have always liked about physics is that you can get by with only a few facts as long as you understand the underlying physics or nature.

3. Science for Development

I am co-chair of a task force for children and school, to assist the work of the International Astronomical Union‘s Office of Astronomy for Development (OAD). This was born out of the International Year of Astronomy in 2009 which reached 815 million people in 148 countries. This showed there was a huge amount of interest for astronomy, particularly in the developing world. The aim of the OAD is to use astronomy to raise the prospects of people in the developing world in 3 areas; children and schools, universities and tertiary education, and the public.

In many areas, providing really good STEM resources has resulted in schools (which in the developing world are often run by the community with teachers who have no training) giving a far better education than if they’d been left to their own devices.

4. Predestination

Macilwain is under the delusion that everyone is born knowing what they like and what is worth spending time at being good at.

‘“I just wish little Mary got the chance to do science at school” is not a phrase, I would submit, that politicians often hear on the doorstep.’

Many children are taught science badly or incorrectly. There are many excellent science teachers out there but there also non-specialists (e.g. biologists teaching physics) or even arts graduates who teach science and who struggle with the material (in UK you do not need a degree in a discipline to be able to teach it to 16 year olds).

The reason kids rarely beg their parents for more science is because the way it is presented in school is often dull and unengaging, and very much about following procedures, largely because of a prescriptive and unimaginative curriculum. At school I hate experiments because you had to blindly follow a list of steps and arrive at an unenthralling result. The chemistry experiments I remember are where things went totally wrong. This is the excitement of science.

STEM activities in schools should be encouraged because there is precious little actually happening in school. Kids want to do more music and more sport because they are engaged in it. They actually make the music. They experience the exhilaration of scoring trys, runs, goals, or roquets. They work as part of a team and are better people for it.

As a professional scientist I experience all of those things with science, but I never once experienced it at school. The science curriculum is lost in the Victorian age of teaching. We do a disservice to kids if we do not find new an innovative ways for them to explore science.

Macilwain suggests you should pursue what you want but not waste time on science because there is no job for you. I would argue the opposite. We do not know what we will end up doing because of changes and chances of this fleeting world. Science is always there because science is about questioning everything and looking for the underlying truth.

Some of the kids in STEM programmes Macilwain wants to stop may turn out to be policy makers, politicians and other people of influence. Would you rather have a scientifically literate person making decisions on stem cell research, GM crops or the nuclear deterrent, or someone who blindly accepts whatever a political aide hands them?

I would be delightfully happy if the end result of me running STEM activities was to stop kids believing science is hard and help them realise that it can be as simple as asking “Yes, but why?”.

So I am proudly on what Macilwain calls  ”a fool’s errand” but hopefully I am more like Ivan the Fool.

Ps. I’ll let you into a secret: STEM activities very often do legitimate, publishable science. See Zooniverse, Faulkes Telescope Project, International Astronomical Search Collaboration, and Las Cumbres Observatory (that’s just a very small sample within the astronomy community).

You are such a Git

I’ve been trying to migrate from using Subversion to Git for my version control. If you need to keep track of the different stages in the development of some work (this usually refers to code but could easily be a paper), you should be using version control.

I’ve been using SVN (often called Subversion) for nearly 10 years and I’m sick of it, so I thought I’d give Git a try. The organization I work for (Las Cumbres Observatory) is a non-profit so we applied for a free organization membership of GitHub (an online repository which will keep all your revision history and has a snazzy web interface). The problem was we have lots of software projects and lots of revision history we didn’t want to lose by giving up on SVN.

Fortunately, lots of people are in this position so there is a project called SVN2Git which will do the hard work of porting all your SVN history over to Git formatted history. It will even change your SVN users to GitHub users for a little more interactivity.

Step 1.

Install svn2git – You’ll need to have “git svn” installed. I’m on a Mac so I had to install Git from Homebrew, not from the .pkg distribution to get svn git.

Step 2.

Create a new directory/folder for the files you will convert from SVN to Git.

Step 3.

Set up an authors file in the correct GitHub format. This will translate your SVN users to GitHub users. You need to have every user who made commits on the SVN project using GitHub. I saved it as authors.txt in the current directory. The format is as follows

Step 4.

You will then run svn2git on your repository (not your local working copy) which will checkout a copy of the whole repository (trunk, branches and tags) and all the revision history. It will then convert all of that into Git format.

Step 5.

Create a repository on GitHub, and tell your newly Git converted project to point at your GitHub repository:

Step 6.

Merge your  newly converted project with the blank repository you created in step 5.

I had to “pull” down a copy of the files from GitHub (which were just the .gitignore and README.md files). Without doing this step I had lots of errors including SSH port 22 fatal error – sounds more alarming that it really was (Git is a bit of drama queen).

Next you will merge the files that svn2git converted for you,

All of your files should have made it into GitHub, complete with committers linked to GitHub users and a full verion history.

Saucy minx

The Double Decker effect

Have you ever had a Cadbury’s Double Decker? For the uninitiated it is a chocolate bar with a layer of nougat (is that pronounced ‘nugget’ or ‘noogar’?) on the top and a layer of rice crispy balls on the bottom, smothered in chocolate. When I get just over half way through one I always experience, what I call, the ‘Double Decker effect’.

The experience starts when you look through the variety of chocolate bars in The Shop. You simply cannot decide which to buy. Obviously it’s going to be either a Lion bar, Wispa, or Kit-kat Chunky but then you realise there are no Lion bars and you really wanted something a bit different. Its been a long time since you’ve had a Double Decker and the wrapper is a rakish combination of orange and purple. You a little bit too long and it has you. You can taste the sumptuous Cadbury’s chocolate. You can feel the crunch of rich crispies. You want it, and looking at it sitting on the shelf in that seductive manner, you can tell that it wants you.

You gently pull it from the roughly torn cardboard pack with all its sisters and look at it sitting in your grasp. It feels right. A bar that will not only satisfy you but make you hungry for more. You pay, and before you have left The Shop, you have subconciously torn the wrapper exposing an indecent amount of chocolate. Stepping into the street you tentatively touch the bar to your lips. You’ve been hurt before and you are nervous about that first bite. Close your eyes and take a modicum of chocolatey, nougatty goodness.

It is delicious! You wonder why you have been eating any other sort of food. This is the finest of all foods. If you could taste colour, this would be a rainbow. You eat and everything is good. Troubles are forgotten and all is calm. A tear trickles down your cheek.

This is not the ‘Double Decker effect’.

You are two thirds of the way through and it begins to rain. There is too much nougat. The sweetness overpowers you and you feel unpleasantly giddy. You masticate a gloopy mess which does not end. Each chew brings forth more of this unending saccharine. At the same time you realise that, next to previously chewed chewing gum, rice crispies are possibly the blandest substance in existence. You would expect the banality of the rice crispies and the potent sweetness of the nougat would cancel each other out. This is where Cadbury’s have been really clever because they on intensify each other’s strength, leading to an unending confectionary nightmare.

By the end you rue the day you ever were seduced into looking at this hate-filled chocolate bar.

This is the ‘Double Decker effect’.

I find this experience can be applied to all manner of things. Things where there is a sufficiently long gap between successive relapses. Like child birth, watching Last of the Summer Wine, drinking real ale on an empty stomach, skiing, listening to Portishead on wet February evening, visiting Coventry, and demonstrating to your wife that you know the name and director of every Star Trek: The Next Generation episode.

You have been warned.

Elemensus box

First appearance of Elemensus

Making words out of Periodic Table elements – We’ve all done it (well a certain subset of us have at any rate). Its a bit like making words on your calculator – you write a couple of them (‘goggles’ and ‘blobs’ were some of the words that had us school boys tittering) and then don’t take it any further. Periodicalised words are similar, you might realise you can make BaBe (out of Barium and Beryllium) or CaPuCCInO (out of Calcium, Plutonium, Carbon, Indium, and Oxygen) but how far can you take it?

I had this conversation with Tony Davis at the Hay Literature Festival in 2008 on the Art Meets Matter stall, whilst buying some of their iconic Penguin mugs. He was interested in the language and vocabulary you can create and I was interested in making something that could churn this vocabulary out automatically. We chatted in dribs and drabs but starting going on this project in earnest towards the end of 2011.

Fast forward to the 2012 Hay Literature Festival. Art Meets Matter is selling a brand new, word-based, periodic table inspired board game which gives you 162 tiles with Periodic Table elements on side and A-Z letters on the other. Its called Elemensus. For more about how you play the game have a look at the Elemensus info page.

It was quite a rush to get the board game finished in time but Tony and I were adamant we wanted it to be launched at Hay. All the graphic design work is down to Tony – he has produced a number of games previously so I knew it was in capable hands (see Penguin Bookchase). I personally think he’s done an amazing job. Elemensus looks beautiful.

All the phone calls, late-night coding, sourcing of astronomical images and tight production deadlines, were worth it when I saw people playing the game. Elemensus was launched on 1 June 2012. I spent the weekend of 8-9 June almost continually playing the game with the punters at the Hay Festival. It was truly amazing, rather like the feeling parents must have when their children grow, leave home and become professional scientists… I felt like a proud dad hearing people really like something that I helped create. I must have played 40 rounds of the game with different mixes of people, from 10 year olds to 70 year olds. Actually the 9-12 year olds seemed to enjoy playing much more than any other age group, which could be because instead of thinking too hard about spelling words letter by letter they were more used to constructing (and deconstructing words) phonetically. If they got a Thorium [Th] tile or Erbium [Er] tile they could effortlessly find loads of words involving the sounds ‘er’ or ‘th’.

Elemensus is just the first in a longer line of products designed around the periodic table, we’ve called Periodic Thinking. We will be publishing a ThEsAuRuS Aug/Sept 2012 with facts, word lists and the full richness of the Periodic Thinking project. Elemensus is the flagship of Periodic Thinking and I am really proud of it.

Haley and I playing the game at home. We played the rule that science words scored double. Fortunately from this angle you can’t see what the science words were…
IMG_0855

Leiden and Universe Awareness

This has been an exciting week of planning. You don’t tend to think of planning as being an exciting process, but this was. I traveled to Leiden in the Netherlands to visit the world famous Pedro Russo. If you were involved in any part of the International Year of astronomy in 2009, you will have had emails beginning “Dear Friends” giving you weekly updates about exciting astronomical events happening in local and remote parts of the world. Pedro now is the international coordinator for the Universe Awareness programme, based in University of Leiden. Recently he and I have taken the reins of an IAU task force for children and schools, under the guidance of the IAU‘s office for astronomy development. The 2 of us and our task force members are tasked with helping to form a strategy which will inspire children and schools in developing countries to engage in science investigations at any level.

It was rather a jolly week all told. I stayed in a very pretty part of Leiden in a huge town house which the owner is exploring turning into a guest house, using the Netherlands “4 bed rule” – where any house can be a guest house as long as you don’t rent more than 4 rooms out. Pedro used his powers to persuasion to get me a bike to use for the week too, which was incredibly handy for getting around. It took some mastering because it had neither brake levers nor gears. You had to back-pedal to brake. I only fell of once or twice. Per journey. I will certainly be taking my Brompton on my next visit.

We spent a very fruitful week discussing plans for our IAU task force (I really think we should have an outfit like superheroes or at the very least a sports strip), created a collaboration between our 2 projects (Las Cumbres Observatory and Universe Awareness), and had a jolly time throwing science outreach ideas around.

I gave a science communication seminar about the citizen science mission of Las Cumbres Observatory on the Wednesday to a packed lecture room (actually, there was quite a lot of space). It seemed to go down well, and I had nice comments from people which always helps. One of the ladies attending had told me that she had seen Brian Cox give a TED talk and she thought he was “hot”, but when she watched Wonders of the Solar System the “hotness” waned, and finally he “was totally out of hotness”. When I saw her texting during my seminar I saw this as an opportunity to acquaint every one with this story and speculate if she was updating the “hotness” of my talk. She told me afterwards, that if I’d had any “hotness” it had all gone now. She also said I was like a “chubby” Pedro. I will leave the final word on that up to the discerning reader…

On my final night we took a trip to the Hague to sample the delights of a local festival, Pop Hot Spot, where we were introduced to the music of a band of 16 years olds called Jungle by Night. Their musical genre was something called Afro beats (which I have never heard of) but reminded of prog-rock music, particularly Yes. Maybe in 40 years they will still be touring and making a comfortable living of the songs they wrote in the youth. I rather enjoyed what they had to offer, particularly where they said that the next song was inspired by Murder, She Wrote. I’m not entirely sure of the root of this inspiration, but you never can tell with these arty types. Maybe there was an off stage murder, or maybe they composed it in Cabot Cove.

17 May was a public holiday in the Netherlands in celebration of the newly regenerated Jesus’ ascension into Heaven, so that mankind can get on with religious wars and prejudice for the foreseeable, without the almighty giving them disapproving looks. The owner of this car is clearly a big fan of Jesus’ mum. They missed a trick. If only they’d had the Blessed Virgin Mary (BVM) stenciled onto a BMW. Notice the dinosaur on the dashboard; they’re clearly not all bad.

The Cardiff-Amsterdam city hopper service by KLM makes it so convenient to pop over the Leiden I hope this is the beginning of more collaborative trips, in both directions.

A word to the wise, there are the most delicious waffles to be had in the vending machine of the Leiden University Physics department. Only 80c too.

Stand up and be scared

Yesterday was the most terrifying Halloween I’ve had for many a long year. It was finally my time to do stand-up comedy at the Bright Club Wales. I had tentatively agreed to do it way back in April when I saw a colleague, Peter Coles, doing a turn. I have had a naive curiosity about doing stand-up and I have always wanted to give it a go. Because I firmly believe that you should never be too scared to give anything a go, I signed up.

I’ve done after-dinner speeches, been best-man at 2 weddings and regularly give public talks so I thought I would be fine. Until I remembered that this time I had to be funny. It wasn’t an option to go on stage and be amusing and sweet in a slightly awkward way. I actually had to make people laugh. Fortunately, it went really well and I had an amazing time on stage. I decided better of doing a Brian Cox impression.

Before you watch be warned there is strong language from the start – it got me a cheap laugh!

There were a couple of other amateur acts including Rhys who used to be a student of mine, and some professional acts; Bob Ramsey, a spirit medium (he looked more like a large to me) and Jon Chase (aka Oort Kuiper), a really brilliant rapper and science communicator. His video of the Periodic Table rap is great.

And lastly, many thanks to MC Dean Burnett for invaluable advice while I was preparing my set.

Astronomers like to MOVE IT

Most of us talk with our hands. I do. Particularly when I’m trying to explain a difficult concept in physics. The first, and most lasting, recollection that one of my best friends has of me is waving my hands around trying to explain some long forgotten point about astronomy.

Whilst editing some video for Teapots from Space, my friend Jon remarked that with the sound off the astronomers looked like they were dancing…..so on a rainy afternoon I explored that a bit more.

I am very sorry to draw your attention to flamboyant moves of Dr Chris North, Dr Rob Simpson, Dr Pete Hargrave and Prof. Peter Coles. Actually, I’m not. They are guilty as charged!

Naked twix

Tea twix tube

This is a quick blog response to Amanda (famous for her Astropixie blog). She is currently observing at the AAT (Anglo-Australian Telescope) and because the weather is exceedingly inclement, they are spending the long, starless nights doing Tim Tam Slams. While I was a PhD student I became a little bit obsessed with doing Tea-twix-tubes or twix-straws. The idea is similar. Jon and I thought we’d show you how it’s done.

First, take your scrumptious Twix finger

Next bite the ends of it (you can see me having a good old chew in the picture). This is essential to getting a good draw of tea through the body of the twix finger. Use the moments after you have bitten the ends of your twix off to go and make a nice hot cuppa, or you might be tempted to gobble the rest of the finger straight off. Don’t try this with coffee and definitely not hot chocolate (unless you want to be bouncing off the walls like a toddler on Haribo). Tea works best. We are British you know.

Dip the end of your twix into the tea and suck quite hard. Don’t be put off by the twix getting a little soggy. You will taste tea through the twix, so if you have really hot tea, its best to put your twix in the fridge for a little bit so you scold your tongue.

Its so delicious you won’t want to stop.

When the chocolate is starting to melt its best to abandon your new straw and munch away at it. You’ll find it is slightly slimy but utterly delicious! Fortunately a twix is more substantial than a Tim-tam so there is little danger of having to fish the soggy remains out of your cup.

Liven up any cup of tea with a Twix Straw!

Autons

First aid and invasion of the autons

This week I thought “Why not do a first aid course?”. The company offered us all a day out at the American Red Cross, to learn CPR (cardiopulmonary resuscitation), basic first aid, and some tips on when to use an AED (automated external defibrilator). It was rather a jolly day, despite being constantly reminded not only of our own mortality, but how ‘dangerous’ CPR could be. Dangerous to the first aider. We saw countless videos where we told to find an antibacterial gauze before you give someone the kiss of life.

Did you know that the person next to you could have AIDS or hepatitis? Its a battle-ground, and you have to protect yourself. So keep your antibacterial gauze handy if you think you might need to snog someone back into the land of the living. Otherwise, you are in danger. Not ‘at risk’, actually ‘in danger’. Just imagine what you could pick up on the bus or, worse yet, in a night club. Be careful when a cashier hands you your change in the supermarket. They might have just sneezed on your money. It turns your blood cold, doesn’t it.

We spent a good portion of the day role playing with these sleepy fellows. Be careful, I suspect they might be Autons

Mine was really poorly. He experienced choking, difficulty breathing, a sudden cardiac arrest and a heart attack. Fortunately I am now trained. I tried a polo mint on him first. He didn’t take to that so I whipped out the AED and zapped him. An AED, if you haven’t had the pleasure, is rather like a Speak-and-spell but with a cheeky little afterbite. Rather disturbingly, a disembodied voice tells you what to do at each step; where to put the pads, how long to wait, if there is a pulse, when its going to shock. The only thing it doesn’t tell you is the cost, but who wants to know that when you’re ill, you might have a heart attack.

The depressing fact is that americans have to pay for every trip in an ambulance – $3000 thank you very much. Republicans are perfectly comfortable with socialised military, police, fireservice and ambulance service. But medicin? Hell no! those illegal immigrants and homosexuals will use my tax-dollars for treating their perverted diseases.

Maybe I’m old fashioned but I don’t want even 30 seconds contemplation of whether I can afford an ambulance trip and hospital stay, if my nearest and dearest is involved in a serious accident. When you put a price tag on it, that is what happens. Its a very Thatcherite principle. If you can afford $1000+ per month, you get insurance. If you can’t you, are treated like a second-class citizen, suffer poor health and end up with huge and unpayable debts.

If you think that paying $100 for a prescription (as Haley recently paid for eye drops) which would be available over the counter at Boots the chemist, in UK for £7, you are stupid or corrupt. Socialized health care is a right not a luxury.