Extracting

About Extracting

After importing, the data can be queried through the command line. For example:

mbpy EXTRACT from-users PRINT

That will produce screens that display the first 5, and last 5 student records, and every column in the student table for those records.

        id      role   oa_id sb_id account_uid
0    10752543    Admin  None  None     None     \
1    10752544    Admin  None  None     None
2    10752545  Advisor  None  None     None
3    10752546    Admin  None  None     None
4    10752547  Advisor  None  None     None      

Scroll down to see more columns for these records. (Notice the \ indicates there is wraparound.)

These commands can be used to just get data displayed, but it becomes interesting when paired with available commands to perform manipulations. For example, to find the distribution of users' domains, we could do the following:

mbpy \
    EXTRACT \
        from-users \
            --fields email \
    INFO \
        --value-count domain \
    extract-on \
        --column email \
        --pattern '^(?P<handle>.+)@(?P<domain>.+)$'

          domain  count    percent
0      eduvo.com    332  97.076023
1      endvo.com      3   0.877193
2      gmail.com      2   0.584795
3        icio.us      1   0.292398
4  mediafire.com      1   0.292398

This school is supposed to have only eduvo.com accounts. Looks like we have a few users with wrong email addresses! Let's find them:

mbpy \
    EXTRACT \
        from-users \
            --fields id,email,role,archived \
    PRINT \
    query \
            --where 'not email.str.endswith("@eduvo.com")'

Let's see what we got:

        id                 email                 role    archived
0    10752543           [e-mail]    Admin     True
34   10752652        [e-mail]  Student    False
50   10752674            [e-mail]    Admin     True
51   10752675         [e-mail]  Student    False
145  10947491              [e-mail]   Parent     True
166  10947512       [e-mail]   Parent    False
167  10947513       [e-mail]   Parent    False
174  10947520         [e-mail]   Parent     True
175  10947521           [e-mail]   Parent     True
530  10947916         [e-mail]  Student     True
531  10947917            [e-mail]  Student     True
...

Let's limit to only those that are active (not archived), and let's give ourselves a link we can click:

mbpy \
    EXTRACT \
        from-users \
            --fields email,profile \
            --where archived b= false \
    PRINT \
    query \
            --where 'not email.str.endswith("@eduvo.com")'

Command-click to load the url in the browser!

               email                                  profile
17     [e-mail]  https://demo.managebac.com/teacher/users/10752652
26      [e-mail]  https://demo.managebac.com/teacher/users/10752675
58    [e-mail]  https://demo.managebac.com/teacher/users/10947512
59    [e-mail]  https://demo.managebac.com/teacher/users/10947513
149               [e-mail]  https://demo.managebac.com/teacher/users/10947922
150       [e-mail]  https://demo.managebac.com/teacher/users/10947923
151          [e-mail]  https://demo.managebac.com/teacher/users/10947924
152          [e-mail]  https://demo.managebac.com/teacher/users/10947925
153             [e-mail]  https://demo.managebac.com/teacher/users/10947926
199              [e-mail]  https://demo.managebac.com/teacher/users/10947975
523       [e-mail]  https://demo.managebac.com/teacher/users/10948377
524          [e-mail]  https://demo.managebac.com/teacher/users/10948386
525          [e-mail]  https://demo.managebac.com/teacher/users/10948389
530  [e-mail]  https://demo.managebac.com/teacher/users/10948406

The structure of these commands are the following:

mbpy <MODE> <extractor/streamer> <LOADER> [<chain> <chain> ...]

Command / subcommand	Explanation	Comment
<MODE>	Either EXTRACT or STREAM	In extraction mode, it loads all the information into memory. In stream mode, it loads by chunks.
<extractor>	from-students, from-classes, etc.	mbpy EXTRACT --help to find full list
<streamer>	See "Streaming" section	mbpy STREAM --help to find full list
<LOADER>	print or pprint or csv	mbpy EXTRACT from-students --help to see full list of Loaders.
<chain>	Peform manipulations after extraction.	Optional.

To learn more about how to query with extractors, continue to Querying Commands:

Querying Commands