1. Qube
2. Qube! Documentation
3. Start Here
4. Monitoring Jobs in the UI

Monitoring Jobs in the UI

Now that you have submitted some jobs, you will want to monitor them in the Qube UI. In fact, there are now two Qube UIs - the original, now called Wrangler View (because it contains features that an administrator or render wrangler would use) and qubeArtistView, intended for artists to monitor and interact with their jobs.! UI.

If you have been working through the Qube 101 documentation in order, you will probably be running the Wrangler View, which has a white background. In this section we are going to skim across the surface of the qubeArtistView Qube! UI. Visit the Qube! UI User's Guide for more information.

Launch the Qube

...

! UI

To launch the Artist ViewQube! UI, do this:

• Windows: Click on the black and white Qube ArtistView desktop icon, or locate it in the PipelineFX->Qube! UI start menu item, or double click it at Program Files\pfx\qube\qubeArtistView folder (Note: not the "Program Files (x86)" folder)QubeUI\QubeUI.bat.
• OS X: Click on the black and white Qube ArtistView desktop icon, or locate it in the Run /Applications/pfx/qube folder/QubeUI.app
• Linux: Run 'qubeArtistView' at the shell prompt (no quotes). By default, qubeArtistView will be located in /usr/local/pfx/qube/qubeArtistViewqubeui/qubeui.

You should see this:

Image RemovedImage Added

You should see the test jobs or other jobs that you have submitted. Notice that they are color coded - if you are lucky, they will all be blue, which is the color of a successfully completed job. Jobs which failed are colored red. The jobs also have a status, such as "complete", "failed" or "running."  Running jobs are shown in green. There are other colors for other statuses, which you will become familiar with as you submit and monitor jobs.

...

The row of buttons just under the Jobs tab is a set of filters which will show or hide jobs in the Jobs Panel. An important one to know about is the User Filter - when set, you will only see your own jobs. When unset, you will see all jobs. Click and unclick this button to see the difference. Note that if you are on Linux, you may have submitted jobs as the root or qubeproxy user which will be hidden if you choose this filter.Image Removed

Image Added

Let's have a look at some simple tasks you can do from this UI. The Jobs pane shows your job(s). By default, they are sorted by Jobid, but you can sort them by any other column by clicking on the column header. For example, you can sort them by Name by clicking on the "Name" label at the top of that column. Clicking once sorts them in ascending order, clicking again sorts them in the other direction. Now, click on one of your test jobs in the Jobs pane.

Image RemovedImage Added

Notice that in the Job Properties panel below the Jobs panel, you can look at the Job Properties on one tab, and the Output on the next two tabs (Output Log & Error Log). This is the Job Details panel - we'll cover that in a second.Image RemovedAlso output in the Stdout Log and StderrLog panels.

Image Added

Interact with a Job

Right-clicking on a job (or a frame, worker, or instance, for that matter), will bring up a context menu:

Image RemovedImage Added

The contents of that menu depend on the entity you clicked.  For example, jobs will have a different list of menu items than will frames or workers.  The menu items are dictated by plugins which can be created or modified.  A more detailed description of the creation or modification of plugins can be found in Adding Custom Plugins.

As of version 67.85-0, the list of available actions are as follows:

• Job
  • Retry Job: Start this job over, using the same job id.  All frames will be re-rendered, regardless of whether or not they had previously completed.
  • Shove: Force the supervisor to reevaluate this job for dispatch. This is a good way to start a job that is stuck in the "pending" state, even though all of its requirements can be met.
  • Retry Failed Frames: Rerender all frames marked as failed
  • Modify: Change modifiable job properties.  When modifying multiple jobs at the same time, you may see <varies...> in the field.  If you wish to append a value, simply add the new value after <varies...>, using the correct delimeter, e.g. <varies..>, global.resource_name=5. If you wish to overwrite all values, simply remove <varies...> and replace it with the new value.
  • Interrupt: Stops Copy IDs to clipboard: Place the job ids of the selected jobs in your copy/paste clipboard.
  • Filter by PGRP: Display only jobs that belong to the selected job's PGRP.
  • Interrupt: Stop the job immediately, and sets the status back to pending.  This is typically used when a high priority job is starved for resources - manually interrupting a lower priority job will free up those resources to allow the high priority job to start.  The lower priority job to start up again as soon as workers become available.
  • Block (Purge): Stops this job immediately, but sets the job state to "blocked".  The job will not run again until it is manually unblocked.
  • Modify Job: Change modifiable job properties.
  • Open log directory: Open the directory containing the job logs for this job. This is only relevant if you're on the worker running the job or supervisor.
  • Prioritize: Change the priority of the selected job.
  • Remove: Stops the job immediately (if running), and permanently removes the job from the qube database.  This cannot be undone.
  • Shove: Force the supervisor to reevaluate this job for dispatch. This is a good way to start a job that is stuck in the "pending" state, even though all of its requirements can be met.
  • Create a zip file of the log directory: Zip the log directory and open the containing folder.
  • Block (Finish Current): Stops this job after the currently rendering frame(s) complete.  The job will remain in a "blocked" state until it is manually unblocked.
  • Block (Purge): Stops this job immediately, but sets the job state to "blocked".  The job will not run again until it is manually unblocked.
  • Unblock: Remove the "blocked" state from the job.  All frames that have not completed will go back to a "pending" state.  Those that have completed or failed will remain completed or failed.
  • Kill Job: Stops this job immediately and sets the state to "killed".  A job that is killed cannot be continued - it can only be retried, resubmitted, or removed.
  • RemovePreempt Job: Stops Preempt the job immediately (if running), and permanently removes the job from the qube database.  This cannot be undone.
  • A/B Wipe in RV: An example of using ArtistView's plugin architecture to perform actions outside of Qube.  When multiple jobs are selected, an RV session will be opened that does a wipe between each job.
  • Browse Output Dirsselected job.
  • Resubmit Job: Submit a new job with the selected job's parameters. A submission dialog is opened so you can modify the job before submission.
  • Retry Failed Frames: Re-render all frames marked as failed
  • Retry Job: Start this job over, using the same job id.  All frames will be re-rendered, regardless of whether or not they had previously completed.
  • Browse Job Output Dir(s): Open a file browser for each output dir in this job.
• Worker
  • Ban host: Ban the selected worker on the supervisor.
  • Partially Lock: Lock a specific number of slots on the selected worker.
  • Ping: Ping the selected workers and retrieve basic information about them.
  • Reboot host: Reboot the selected workers.
  • Shut down host: Shut the selected workers down.
  • Lock Host (Finish Current): Same as above, but the currently rendering frames are allowed to finish.
  • Lock Host (Purge): Stop all jobs running on this worker immediately, then leave the worker in a "locked" state which will prevent it from accepting any more work until it is manually unlocked.
  • Lock Host (Finish Current): Same as above, but the currently rendering frames are allowed to finish.
  • Unlock Host: Remove a worker lock.
  • Remove: Remove the selected workers from the Qube! farm.
• Frame
  • Retry: Rerender Select worker: Display the Workers Panel and select the worker than rendered this Frame.
  • Complete: Change the state of this frame to "complete," even if it has not yet run.
  • Retry: Re-render the selected frame(s)
  • Block: Set the frame's state to "blocked".  This will prevent the frame from being rendering until it (or the job) is manually unblocked.
  • Unblock: Remove the block from the frame, setting it back to a pending state.
  • Kill: Stop this from immediately and set the state to "killed".
  • Complete: Change the state of this frame to "complete," even if it has not yet run.
• InstanceMigrateInstance
  • Select worker: Display the Workers Panel and select the worker than ran this instance.
  • Block: Stop this instance from processing work and set its state back to "pendingblocked".  Default supervisor settings dictate that the next time this instance starts, it will start on a different machine.Retry: Restart this instance, regardless of its current state This instance will not process any more work until it is manually unblocked or the job is retried.
  • Unblock: Remove a block from an instance.
  • Complete: Mark this instance as complete.
  • Interrupt: Stop this instance from processing any more work, but set its state back to "pending".  As with a job's "interrupt" this is typically done to manually free up resources that could be used by higher priority jobs.
  • BlockMigrate: Stop this instance from processing work and set its state back to "blockedpending".  This instance will not process any more work until it is manually unblocked or the job is retried.
  • Unblock: Remove a block from an instance.

• •  Default supervisor settings dictate that the next time this instance starts, it will start on a different machine.
  • Retry: Restart this instance, regardless of its current state.
    
    

Watching Progress

Once a job is running, if it takes more than a few seconds (and most do) then you will want to keep an eye on progress. By default, the UI does not refresh itself, meaning that as the job progresses, nothing will change in the interfaceThe UI will display changes to Jobs and Workers in real-time but frames and instances are only loaded when a job is selected. To update the frame and instance lists double click on the job. You can force an update by simply clicking on the job. You can also refresh all jobs or frames or instances or hosts by clicking the "Refresh" button associated with their list.

You can also configure Qube to automatically update the UI every so often. To do this, go to Edit in the top menu and choose Preferences. Click on Auto Refresh / Interval. You can leave the refresh interval at 60 seconds, or make it shorter or longer.

Image Removed

Note that asking the Supervisor for an update every few seconds puts more load on it, and if there are many artists all doing the same thing, you could be slowing down the overall farm just for the sake of convenience. Think twice before asking for lots of updates you aren't going to be noticing anyway.

Looking at Job Details

The Job Details panel has a number of tabs. The first one, Job Properties, lists a number of attributes of the job, including the frame range and the command and/or scene file used.

The next two are useful for finding out what went wrong with jobs that fail (or getting logging information from jobs that succeed). Most output will go to the first tab, Output Log. Some programs will send error messages and other information to a separate stream, which goes to the Error Log. Scroll to the bottom of either of these - error messages are coded in red, warnings in yellow and info (as determined by the program being run) in green.  All of the tabs and their uses are explained in more detail in the User's GuideDue to network interruption it is possible for some jobs to become out of sync, if you suspect this has happened click the sync button shown in 1 below.

Image Added

Use the sync button sparingly as it can place undue load on the supervisor on large farms.

Looking at Job Properties/Internals and Output

The Job Properties Panel lists basic attributes of the job. The Job Internals Panel shows more in-depth information such as the job package, callbacks in use and the job history.

The output and error of the job's rendering process can be viewed in the Stdout Log and Stderr Log Panels. You can review the image output in the Thumbnails Panel and Preview Panels. Check the User's Guide for the Qube! UI for more information on how to use these panels.

Submitting New Jobs

The Artist View Qube! UI can be used to submit jobs to Qube, ! through the Submit menu item at the top. However, in most cases you will be setting Qube up to allow submission directly from your application, such as Maya or 3dsMax. Submitting directly from applications was covered in the previous section. However, you can still use the UI to submit test jobs, to make sure that things are still working as expected, and to submit jobs for applications where you haven't installed the inApp interface.

The Frames Panel

The right side of the Artist View UI contains the Frames panel, which lists information about all the frames in the selected job. Notice that it has a separate refresh button, which allows you to query the status of just those frames. You can also right-mouse on any of the frames and kill them, retry them, suspend (block) them (and then unblock them), as well as marking them complete.