Showing posts with label api. Show all posts
Showing posts with label api. Show all posts

Friday, April 27, 2012

Automatically Creating Google Homework Sites

We had an interesting Google Sites idea for teaching get floated yesterday....

Q. Could you automatically create a Google Site for a list of students that only they and their tutor can see? Also, at a given time could the students' permission be changed from "writer" to "reader" when the deadline had arrived? It'd be good if you could use a Template Site so that the site could be set up with the right pages and prompts to begin with.


With the GData API it is possible. Here's an example python script that shows how... First, load the libraries. You may need to download these if you don't have them already.


import atom.data
import gdata.sites.client
import gdata.sites.data
from time import sleep


I then created lots of stub functions in the hope that I could better understand how the API works. The problem ( for me ) here was that ...

a. This code uses a now old fashioned means of connecting to the API. If this was going to be "real code" we would have to use OAuth to connect to the API.
b. The API documentation is hopeless
c. It seems to work around the idea of (Atom) feeds with have <entry> objects, which have <other objects> in them, which you get to via various URLs, which just makes the whole process more cumbersome... anyway...

The first thing you do is connect to the API


client = gdata.sites.client.SitesClient(source='University of York', site='UoY') # Can be anything, I think
client.domain = "york.ac.uk" #Not needed for non Apps domains
client.ClientLogin('****@york.ac.uk', '*********', client.source)


... then I wanted to see a list of all my sites and also show the URLs I'd need to use for each.



def list_sites( ):
'''
Shows a list of URLs you will need
'''
feed = client.GetSiteFeed()
for entry in feed.entry:
print '%s (%s)' % (entry.title.text, entry.site_name.text)
if entry.summary.text:
print 'description: ' + entry.summary.text
print "\turl:", entry.find_self_link()
print "\tedit link", entry.find_edit_link()
print "\tacl_link:", entry.get_acl_link().href
print "\tclickable link:", entry.get_alternate_link().href
print "_" * 80

... this produces something like this...




Code Example: Timed Content Creation (code-experiments)
description: This site is an example that uses an App Script to grab some data from the web and save it in a Site's pages
url: https://sites.google.com/feeds/site/york.ac.uk/code-experiments
edit link https://sites.google.com/feeds/site/york.ac.uk/code-experiments
acl_link: https://sites.google.com/feeds/acl/site/york.ac.uk/code-experiments
clickable link: https://sites.google.com/a/york.ac.uk/code-experiments/
________________________________________________________________________________
Collaborative Tools Project (collaborative-tools-project)
url: https://sites.google.com/feeds/site/york.ac.uk/collaborative-tools-project
edit link https://sites.google.com/feeds/site/york.ac.uk/collaborative-tools-project
acl_link: https://sites.google.com/feeds/acl/site/york.ac.uk/collaborative-tools-project
clickable link: https://sites.google.com/a/york.ac.uk/collaborative-tools-project/
________________________________________________________________________________
Collaboratomatic (collaboratomatic)
url: https://sites.google.com/feeds/site/york.ac.uk/collaboratomatic
edit link https://sites.google.com/feeds/site/york.ac.uk/collaboratomatic
acl_link: https://sites.google.com/feeds/acl/site/york.ac.uk/collaboratomatic
clickable link: https://sites.google.com/a/york.ac.uk/collaboratomatic/
________________________________________________________________________________
Departmental 20:20s (departmental-20-20s-july-2011)
description: A place to collect resources from the Information Directorate 20:20 presentations
url: https://sites.google.com/feeds/site/york.ac.uk/departmental-20-20s-july-2011
edit link https://sites.google.com/feeds/site/york.ac.uk/departmental-20-20s-july-2011
acl_link: https://sites.google.com/feeds/acl/site/york.ac.uk/departmental-20-20s-july-2011
clickable link: https://sites.google.com/a/york.ac.uk/departmental-20-20s-july-2011/



With this list of URLs I can then either access a site for editing, or for changing its permissions ( or ACLs) like this....




def get_site(site_feed_url):
'returns a GDEntry site object that you can fiddle with'
site = client.GetEntry(site_feed_url)
print site.title.text
print "\tedit_link:", site.get_edit_link().href
print "\tacl_link:", site.find_acl_link()
return site



def get_site_by_name(name):
feed = client.GetSiteFeed()

for entry in feed.entry:
if entry.title.text == name:
print "url:", entry.find_self_link()
print "edit link", entry.find_edit_link()
print "acl_link:", entry.get_acl_link().href
return entry
break


>>> site = get_site_by_name("My Template Site")

url: https://sites.google.com/feeds/site/york.ac.uk/my-template-site
edit link https://sites.google.com/feeds/site/york.ac.uk/my-template-site
acl_link: https://sites.google.com/feeds/acl/site/york.ac.uk/my-template-site
<gdata.sites.data.SiteEntry object at 0x10143c1d0>

I can then nosey around in my site to see what it can do...


>>> dir( site )
['FindAclLink', 'FindAlternateLink', 'FindChildren', 'FindEditLink', 'FindEditMediaLink', 'FindExtensions', 'FindFeedLink', 'FindHtmlLink', 'FindLicenseLink', 'FindMediaLink', 'FindNextLink', 'FindPostLink', 'FindPreviousLink', 'FindSelfLink', 'FindSourceLink', 'FindUrl', 'GetAclLink', 'GetAlternateLink', 'GetAttributes', 'GetEditLink', 'GetEditMediaLink', 'GetElements', 'GetFeedLink', 'GetHtmlLink', 'GetId', 'GetLicenseLink', 'GetLink', 'GetNextLink', 'GetPostLink', 'GetPreviousLink', 'GetSelfLink', 'IsMedia', 'ToString', '_XmlElement__get_extension_attributes', '_XmlElement__get_extension_elements', '_XmlElement__set_extension_attributes', '_XmlElement__set_extension_elements', '__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_attach_members', '_become_child', '_get_namespace', '_get_rules', '_get_tag', '_harvest_tree', '_list_xml_members', '_members', '_other_attributes', '_other_elements', '_qname', '_rule_set', '_set_namespace', '_set_tag', '_to_tree', 'attributes', 'author', 'category', 'children', 'content', 'contributor', 'control', 'etag', 'extension_attributes', 'extension_elements', 'find_acl_link', 'find_alternate_link', 'find_edit_link', 'find_edit_media_link', 'find_feed_link', 'find_html_link', 'find_license_link', 'find_media_link', 'find_next_link', 'find_post_link', 'find_previous_link', 'find_self_link', 'find_source_link', 'find_url', 'get_acl_link', 'get_alternate_link', 'get_attributes', 'get_edit_link', 'get_edit_media_link', 'get_elements', 'get_feed_link', 'get_html_link', 'get_id', 'get_license_link', 'get_link', 'get_next_link', 'get_post_link', 'get_previous_link', 'get_self_link', 'id', 'is_media', 'link', 'namespace', 'published', 'rights', 'site_name', 'source', 'summary', 'tag', 'text', 'theme', 'title', 'to_string', 'updated']


From here I created a few more functions, mainly just trying to understand how the API works.

def create_site(title, description='', theme='slate', source_site_url=None):
if source_site_url:
#template_site = get_site(source_site_url)
#source_url = template_site.GetId()
print "copying:", source_site_url
site = client.CreateSite(title, description=description,theme=theme, source_site=source_site_url)
else:
#Create a blank site
site = client.CreateSite(title, description=description, theme=theme)
print "site created!", site.find_edit_link()
return site

def get_acls(site_acl_url):
'''
Get the site's permissions as a feed if given an acl_link, like this
'https://sites.google.com/feeds/acl/site/york.ac.uk/my-lovely-site'
'''
if "/acl/" not in site_acl_url:
print "That's probably not the right URL!"
feed = client.GetAclFeed(site_acl_url)
for entry in feed.entry:
try:
print '%s (%s) - %s' % (entry.scope.value, entry.scope.type, entry.role.value)
except Exception, err:
#print err
pass
return feed
def share_a_site_with(acl_url, email, role='writer'):
'''
Needs an acl_link to work
role can be reader, writer or owner
'''
#feed = client.GetAclFeed(acl_url)

scope = gdata.acl.data.AclScope(value=email, type='user')
role = gdata.acl.data.AclRole(value=role)
acl = gdata.acl.data.AclEntry(scope=scope, role=role)
#post(self, entry, uri, auth_token=None, converter=None, desired_class=None, **kwargs) method of gdata.sites.client.SitesClient instance
#fudgetastic!
acl_url = acl_url.replace('https://sites.google.com', '') #!!!! ??? 
acl_entry = client.Post(acl, acl_url)
print "%s %s added as a %s" % (acl_entry.scope.type, acl_entry.scope.value, acl_entry.role.value)

def list_a_sites_acls(site_url):
feed = client.GetAclFeed(site_url)
for entry in feed.entry:
  print '%s (%s) - %s' % (entry.scope.value, entry.scope.type, entry.role.value)
return feed

def list_all_copies(site_url):
''' 
Unfinished: You will need to be able to get all copies of a template site. You can iterate
through all sites and find the sites whose source is a another_site's URL.
'''
site = get_site(site_url)
#get all my sites
#for each, check site.FindSourceLink() to see if it is site_url
def remove_a_user(site_url, email):
"Shows how to remove a users access to a site"
feed = client.GetAclFeed(site_url)
for entry in feed.entry:
print entry.scope.value
if entry.scope.value == email:
print "DELETING ACCESS FOR: ", email
client.Delete(entry.GetEditLink().href, force=True)
break
print "done!"

And Finally....

I made a test function to get my Template Google Site and make copies ( each with their own name ) and allow "writer" access to them based on a list of emails.

def do_test():
template_site_url = 'https://sites.google.com/feeds/site/york.ac.uk/my-template-site'
#This might be read in from a spreadsheet.
students_emails = ['student1@york.ac.uk',
'student2@york.ac.uk', 
'student3@york.ac.uk',
'student4@york.ac.uk',]
for i,student_email in enumerate(students_emails):
#Create a copy of the template site with a new name
client = gdata.sites.client.SitesClient(source='University of York', site='UoY')
client.domain = "york.ac.uk"
client.ClientLogin('****@york.ac.uk', '********', client.source)
new_site_name = "Auntie Homework: %s" % student_email
print "creating:", new_site_name
site = create_site(new_site_name, source_site_url=template_site_url)
sleep(5)
#Get its edit_url
new_acl_url = site.get_acl_link().href
print "sharing:", new_acl_url
#Now add that user as a writer to the new site's permissions
share_a_site_with(new_acl_url, student_email, role="writer")
sleep(5)
print "Done creating", i+1, "Google Sites"


Conclusions

It worked! It made 4 new Google Sites based on my template site, and each was shared with one other person with "writer" access.

But there's still a heap of work to make this into something genuinely useful.

  • Because the client.ClientLogin() method, used to connect to API is deprecated, we will have to re-work this bit, and create an online user interface for it. I think greating a GUI in Google Spreadsheets might be the easiest but my JavaScript aint that hot.
  • Although each copied site keeps a track of the site it was copied from, there's no way of running this script for two separate homework assignments for separate lists of students AND have it keep track of which was in which cohort. 
  • I didn't look at "group permissions" which might be handy. When the homework deadline is passed a student group, rather the writer of the site, might be given read-only access. This would mean that, once finished everyone else can see everyone else's sites. 
  • I found that without the sleep() and re-initialising the client instance each time I wanted to create a new site, that the do_test() function threw a wobbly. 
  • A site can't be created if one with that name/URL already exists. I'd need to create a function to clear all copies of a template site.
  • Potentially, I think we may be able to either subscribe to the newly created sites, or do something with the newly created sites' content feed, so that as a tutor you'd be able to keep an eye on who was or wasn't completing their homework in one place. I didn't get that far though. At very least, the script might create a page on the Template Site that listed the copies that had been made.






















Tuesday, April 24, 2012

Scraping The Festival of Ideas, June 2012

I noticed something on Twitter about the University's Festival of Ideas and thought I'd take a look at the events listing. Not long ago, the Web Office used to put microformat information in web pages so that I could easily add events to my calendar... Either they've stopped doing that, or it's stopped working, so I thought how easy would it be to grab the events listed and add them to my (or a separate calendar).

In order to do this, I'd need to...

  1. Scrape the HTML from the web page and find the event data
  2. Connect to Google Calendar and add the events found


Because I like programming in python, the first thing I did was to go get the latest copy of BeautifulSoup, which is a library that is unbelievably handy for scraping data out of HTML and also Google GData which lets me talk to Google Calendar.

I so I began...


import urllib, urlparse, gdata, time, datetime
from bs4 import BeautifulSoup
import atom
import gdata.calendar
import gdata.calendar.service

... and loaded the libraries.  Then I connected to Google Calendar, like this...


print "Connecting to Google Calendar"
calendar_service = gdata.calendar.service.CalendarService()
calendar_service.email = '*********@york.ac.uk'
calendar_service.password = '**********'
calendar_service.source = 'Google-Calendar_Python_Sample-1.0'
calendar_service.ProgrammaticLogin()



 .... then got the web page with the Festival of Ideas events on it like this...


url = 'http://yorkfestivalofideas.com/talks/'
print "reading ", url
u = urllib.urlopen( url )
html = u.read()

... At this point, I knew I wanted to create a separate calendar, so I made one in Google Calendar ( IMPORTANT! Set the timezone of your newly created calendar!!! ). Once I'd done this, I could then find what's called the calendar link which you use to specify which calendar you want events to go into...


def get_my_calendars_url(cal_name):
feed = calendar_service.GetOwnCalendarsFeed()
for i, a_calendar in enumerate(feed.entry):
name = a_calendar.title.text
print i, a_calendar.title.text, a_calendar.link[0].href
if name == cal_name:
return a_calendar.link[0].href


calendar_link = get_my_calendars_url("Festival of Ideas")




So, now I have some HTML with useful information in it and a way of connecting to my chosen calendar... I need to use Beautiful soup to fish out the data I need.  I begin like this...


soup = BeautifulSoup( html )
events = soup.find_all("div", {'class':'event'})


... Now the HTML has been turned into a "soup" which means I can do fancy things with it... like the 2nd line above where I grab any DIV that is of class "event" from code that looks like this..


<div class="event">
<div class="eventdate">
<div class="day">
Thu
</div>
<div class="date">
14
</div>
<div class="month">
Jun
</div>
</div>
<div class="eventdetails">
<p class="eventtitle">
<a href="/talks/2012/frenck/">
Where it all began: The Big Bang
</a>
</p>
<p class="eventteaser">
Professor Carlos Frenk will open this year's York Festival of Ideas with a talk on the biggest metamorphosis of all - that of the universe as a whole, from the simplicity of the Big Bang to the complexity of the universe of galaxies, stars, and the planet on which we live.
</p>
</div>
<div class="clear"></div>


...Once I've got a list of events I can then do this... which finds the title, and the text and the dates and times of the events....



for event in events:
try:
title = event.find('p', {'class':'eventtitle'}).find('a').contents[0].strip()
href = event.find('p', {'class':'eventtitle'}).find('a')['href']
href = urlparse.urljoin(url, href)

#Get the actual page in the href!
u = urllib.urlopen( href )
event_html = u.read()
small_soup = BeautifulSoup(event_html)
start_time = small_soup.find('abbr', {'class':'dtstart'})['title']
st = time.strptime(start_time, "%Y-%m-%dT%H:%M")
end_dt = datetime.datetime(2012, st.tm_mon, st.tm_mday, st.tm_hour+2, 0, 0)
end_time = end_dt.strftime("%Y-%m-%dT%H:%M:%S")
start_time = start_time + ":00" #HACK UG!


teaser = event.find('p', {'class':'eventteaser'}).contents[0].strip()
teaser =  teaser + "\n\n" + href

print "creating event:", title
print create_event(title, teaser, "York, UK", start_time, end_time) 

print "_" * 80
except Exception, err:
print err


.... and the create_event code, which uses that calendar_link mentioned earlier, is...


def create_event( title='A lovely event', 
    content='Some text about it', 
    where='York, UK', start_time=None, end_time=None):


    event = gdata.calendar.CalendarEventEntry()
    event.title = atom.Title(text=title)
    event.content = atom.Content(text=content)
    
    #time_zone = 'Europe/London'
    #event.timezone = gdata.calendar.data.TimeZoneProperty(value=time_zone)
    event.where.append(gdata.calendar.Where(value_string=where))


    if start_time is None:
      # Use current time for the start_time and have the event last 1 hour
      start_time = time.strftime('%Y-%m-%dT%H:%M:%S.000Z', time.gmtime())
      end_time = time.strftime('%Y-%m-%dT%H:%M:%S.000Z', time.gmtime(time.time() + 3600))
    event.when.append(gdata.calendar.When(start_time=start_time, end_time=end_time))


    new_event = calendar_service.InsertEvent(event, calendar_link)


    return new_event



... Putting it all together I got a events that can be displayed in a fairly rubbishy widget ( go to June 2012 to see the events! ) or a calendar that anyone can browse here.

https://www.google.com/calendar/embed?src=york.ac.uk_9d9et5aruukobiaqpgke4n63rk@group.calendar.google.com&ctz=Europe/London&gsessionid=OK









The End Result?


To be honest, presentation isn't Google Calendar's strongpoint is it? It's fugly. It's all about the utility though... and I suppose making sure you get to those events.

I guess my point was, and is, that more of this sort of data should be ending up in places that I can use it, i.e in Google Calendar rather than hiding on a web page somewhere. Maybe this little bit of code will help someone to get their events in a more usable form.



 

© 2013 Klick Dev. All rights resevered.

Back To Top