research software engineering with python ucl

Advanced Research Computing

Course Catalogue

This page provides a summary of the research IT training we provide for UCL staff and students as well as some of our recommendations for online courses from other providers.

ARC courses

Electronic research notebook (ern) training, digital skills development courses, recommended online courses.

• External training providers

Follow us on Twitter (@ucl_arc)  or sign up to our mailing list  for announcements about research IT training opportunities at UCL and elsewhere.

An introduction to command line interaction with the Unix shell.*

Available on Moodle  

* Software Carpentry Workshops and the HPC Carpentry for UCL Central Cllusters course also teach the basics of using the shell.

This course will introduce fundamental programming concepts and is aimed at researchers who want to use Python to manipulate and analyse data.

Available on Moodle

This course introduces the essential things you'll need to know in order to access UCL's research computing clusters, use the command line, transfer files, load pre-installed software packages using environment modules, and submit jobs on the cluster using a scheduler.

Next instance: to be confirmed Further information and registration:  UCL HPC Carpentry

Software Carpentry  aims to help researchers get their work done in less time and with less pain by teaching basic research computing skills. This hands-on workshop will cover concepts and tools, including program design, version control, data management, and task automation. Participants will be encouraged to help one another and to apply what they have learned to their own research problems.

Next instance: to be confirmed

Complete our expression of interest form

if you would like to be notified of future classroom-based instances of these courses when they become available (most of the courses are also available as a self-paced option on Moodle). You can also use the form to suggest topics you would like us to consider covering in future.

UCL's new ERN service is provided by RSpace, who have some useful online training videos. Links to these can be found on the ERN further information  web page.

ISD's Digital Skills Development team also runs courses that may be of interest to researchers; forthcoming courses include:

22nd Nov 2022, 12-1pm DSD: In a Nutshell: Git version control (remotely-taught session). New date

This "in a nutshell" training is an introduction to setting up version control using Git. The command line interface (in contrast to the Graphical User Interface, or GUI) will be used to understand key functions of Git. This is a "hands on" workshop; prior to attending the session, participants are required to set up their computers for command line using Linux. (Further information on what you need to do will be sent to you when you register). Having done this course, participants should continue further training; we recommend the Software Carpentry Workshop (see ARC courses, above).

16th & 23rd Nov 2022 DSD: An Introduction to R with Rstudio (campus-based; 2 x three-hour in person sessions)

This course is designed to help you to use R and R-studio to import, manipulate, describe and analyse categorical and continuous data and to produce plots and tables to support analysis. You should have a good understanding of statistical concepts. No prior experience with R is required, but experience of using a spreadsheet application such as Excel would be an advantage.

Please follow the links to find out more and register if interested.

In the 2020-21 academic year, we ran a series of lunchtime webinars for new researchers (and also for those who were just new to our services!) which lasted approximately 45 mins including questions. Recordings of these webinars are available below.

See  Tools and Tips  for more tools, tips and training resources recommended by members of the ARC team.

Other research IT training providers

ARCHER2 is a national UK supercomputing service which also provides a range of training courses at sites around the UK which are free for UK academics. See their  training page  for details.

LinkedIn Learning

LinkedIn Learning provides video tutorials covering a wide range of IT skills including programming, data analytics, web development, data visualisation, continuous deployment and more – all free to UCL staff and currently enrolled students. See  ISD Learning & Teaching Services  for details.

UK Data Service

As well as providing access to a range of social and economic data, the UK Data Service run regular  training events and webinars  covering a wide range of tools and methods for working with qualitative and quantitative data. See the  UK Data Service YouTube channel  for recordings of past webinars.

• Prerequisites
• Experiences

Assessments

• Software Projects

Module: Software Projects

Learning outcomes, software projects learning outcomes.

By the end of this module students should be able to:

• Understand and communicate the benefits of libraries to a research programmer
• Evaluate the quality of third-party libraries
• Install python packages using pip, conda and other sources
• Organise python code into a structured package
• Use argparse to handle command line arguments to executable scripts
• Understand and communicate the value of documentation and use appropriate tools to create documentation
• Discuss the pros and cons of different software design and development methodologies
• Understand software licenses, including how to select an appropriate language for a given project

Introduction to libraries

Writing Libraries

How to write a library

What is Argparse and how to use it

Introduction to packaging

Documentation

Introduction to documentation

Software Project Management

Introduction to software project management

Managing software issues

Introduction to issues and issue tracking

Software Licensing

Introduction to software licensing

Experiential Learning

Solution: energy example.

energy exercise solution

Practicals - Software Projects

Software projects quiz.

Return to Modules

Centre for Advanced Research Computing

ARC is UCL's research, innovation and service centre for the tools, practices and systems that enable computational science and digital scholarship

Archive for the 'Python' Category

• Randomising Blender scene properties for semi-automated data generation

By Ruaridh Gollifer, on 12 December 2023

Blender is a free and open-source software for 3D geometry rendering . Uses include modelling, simulation, animation, virtual reality applications, and more recently synthetic datasets generation . This last application is of particular interest in the field of medical imaging, where often there is limited real data that can be used to train machine learning models. By creating large amounts of synthetic but realistic data, we can improve the performance of models in tasks such as polyp detection in image guided surgery . S ynthetic data generation has other advantages since using tools like Blender gives us more control and we can generate a variety of ground truth data from segmentation masks to optic flow fields, which in real data would be very challenging to generate or would involve extensive time consuming manual labelling . Another advantage of this approach is that often we can easily scale up our synthetic datasets by randomising parameters of the modelled 3D geometry . There can be challenges to make the data realistic and representative of the real data .  

The Problem  

The aim was to develop an add-on that would help researchers and medical imaging experts determine which range of parameter values make realistic synthetic images. Prior to the project, the dataset generation involved a more laborious process of manually creating scenes in Blender with parameters changed manually for introducing variation in the datasets. A more efficient process was needed during the prototyping of synthetic dataset generation to decide w hat range of parameters make sense visually , and therefore in the future, to more easily extend to other use cases .

What we did  

In collaboration with the UCL Wellcome / EPSRC Centre for Interventional and Surgical Sciences ( WEISS ) , research software engineers from ARC have developed a Blender add-on to randomise relevant parameters for the generation of datasets for polyp detection within the colon . The add-on was originally developed to render a highly diverse and (near) photo-realistic synthetic dataset of laparoscopic surgery camera views. To replicate the different camera positions used in surgery as well as the shape and appearance of the tissues, we focused on randomising three main components of the scene: camera transforms (camera orientation and location), geometry and materials. However, we allowed for more flexibility beyond these 3 main groups of parameters, implementing utilities to randomise other user-defined properties. The software also allows the following features: 1) setting the minimum and maximum bounds through an input file, 2) setting a randomisation seed for reproducibility, 3) exporting output parameters for a chosen number of frames to an output file. The add-on includes testing through Pytest , documentation for users and developers , example input and output files and a sample Blender scene .

The outcomes  

Version 1.0.0 of the Blender Randomiser is available under a BSD 3-Clause License. The GitHub repo is public  where the software can be downloaded and installed with instructions provided on how to use the add-on. Examples of what can be produced in Blender can be found at the UCL Research Data Repository (N.B. these examples were produced manually prior to completion of this project).

Developer notes are also available to allow contributions.  

Sofia Minano and Ruaridh Gollifer

Filed under Collaboration , Medical imaging , Python

Tags: machine vision , python , research-software , surgery

Comments Off on Randomising Blender scene properties for semi-automated data generation

Recent Posts

• Role models, outreach and changing the future of STE(A)M
• k-Plan now available to researchers!
• First Julia workshop at ARC
• Using continuous integration efficiently
• December 2023
• November 2023
• October 2023
• September 2023
• October 2022
• February 2022
• December 2021
• October 2021
• September 2021
• November 2020
• January 2020
• August 2019
• January 2019
• September 2018
• December 2017
• January 2017
• September 2015
• February 2015
• January 2015
• December 2014
• October 2014
• August 2014
• January 2014
• November 2013
• October 2013
• August 2013
• February 2013
• January 2013
• December 2012
• November 2012
• October 2012
• Collaboration
• GitHub Actions
• GitLab CI/CD
• Medical imaging
• Recruitment
• Technologies
• Uncategorized
• War stories
• Entries RSS
• Comments RSS

Research Software Engineering with Python

Download notes as PDF .

Powered by Jupyter Book .

• suggest edit

1. Introduction to Python

1. introduction to python #.

Why use scripting languages?

Python and the Jupyter notebook

Variables, using functions, and types

Loops and control

Data structures: lists, dictionaries, and sets.

1.0 Introduction to Python (10 minutes)

1.1 Variables (20 minutes)

1.2 Functions (20 minutes)

1.3 Types (20 minutes)

1.4 Containers (10 minutes)

1.5 Dictionaries (10 minutes)

1.6 Data Structures (5 minutes)

1.7 Control and Flow (15 minutes)

1.8 Iteration (10 minutes)

Total time: 2 hrs

Exercises #

Classroom exercises are grouped together at the end of the module: 1.9 Classroom Exercises . Each exercise is labelled with any sections whose contents are relevant. We recommend that instructors schedule the exercises to be done in groups during breaks in the taught content. However, it is important that participants also have some time away from their screens. Exercises can also be left as self-paced homework assignments if preferred.

• ReadingLists@UCL Help

Browse Hierarchy COMP0233: COMP0233: Research Software Engineering with Python

Lists linked to comp0233: research software engineering with python, add list to this module, add existing node.

Python Packages §

We provide a collection of installed Python packages for each minor version of Python, as a bundle module . This page lists the packages for the current recommended Python 3 bundle.

This can be loaded using:

The version of Python 3 provided with this bundle is currently Python 3.9.10.

Note that some packages we do not provide this way, because they have complicated non-Python dependencies. These are usually provided using the normal application modules system. This includes TensorFlow .

The following list was last updated at 11:05:14 (+0000) on 03 Mar 2024.

Search code, repositories, users, issues, pull requests...

Provide feedback.

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly.

To see all available qualifiers, see our documentation .

• Notifications

MPHYM001/G001 - Research Software Engineering with Python Coursework II (UCL, 2017-18)

SSabas/ucl-softeng-cw2

Folders and files, repository files navigation, refactoring trees, an exercise for a course in research software development in python, at ucl..

In this exercise, we will convert badly written code, provided here, into better-written code. We will do this not through simply writing better code, but by taking a refactoring approach, as discussed in the lectures. As such, we use of git version control, to make a commit after each step of the refactoring, with a commit message which indicates the refactoring you took, will be critical to success.

• Python 100.0%