Limitations

The vast majority of functionality available from the AMT API is supported. However, a few things are not supported, or are have only limited support:

Missing capability Note
ChangeHITTypeOfHIT Rarely useful
SetHITAsRevewing Not useful
You cannot set the HITLayout parameter when creating a HIT type Not useful, except for working with HITs that were designed on the AMT web site.
Notification receptors Support for notifications is in incomplete and may never be completed. According to the AWS documentation, notifications do not take effect until a few minutes after AWS receives the request to start them, and even when working they may be batched and/or have a delay up to a few minutes. CrowdLib previously supported notifications by HTTP and email, but not Amazon’s Simple Queue Service (SQS). Notification support has not been recently tested and may be broken.
Shortcuts for the built-in question fields cover only very basic options. For example, the OtherSelection option of multiple-choice questions is not supported. This could be added in the future if there is any interest at all. In practice, I find the built-in question fields to be good for testing and simple demonstrations, but not useful for anything real. At any rate, you can still access the full functionality by providing a QuestionForm XML structure, instead of using the shortcuts.
DisposeHIT To make a HIT unavailable to workers, we simply force it to expire. We know of no benefit to deleting it from AMT’s servers. However, deleting a HIT prevents you from fetching it from another location (e.g., another CrowdLib installation), and prevents you from un-rejecting a mistakenly rejected HIT.
GetRequesterWorkerStatistics CrowdLib worker’s approval rate with you is calculated by CrowdLib using Worker.approval_rate().
Advanced options for custom qualification types CrowdLib does support all of the built-in qualification types (e.g., approval rate, country, age, masters, etc.), as well as qualification types that are automatically scored using an answer key. However, it does not support directly assigning/revoking/getting qualifications, or any operations related to qualification requests.

Concurrency

CrowdLib has been used successfully in a multi-threaded web application with some occassional concurrent access. It can normally be regarded as thread-safe with two conditions:

  • Changes to settings should be made from only one thread. Otherwise, behavior might be unpredictable.
  • The CrowdLib database should be located on the same machine that executes the code, i.e., not accessed by NFS. Otherwise the database could be corrupted. This is also an issue for applications with multiple concurrent processes. The SQLite FAQ gives more detail.

These restrictions do not apply to single-threaded, single-process applications or to internal web applications with light load (i.e., accessed by only one person at a time).