CategoryGoogle Summer of Code

GSoC Fedora Happines Packets Update – Week 8th and 9th

The talk got accepted a month ago (https://pagure.io/flock/issue/55) but I didn’t know if I’d be able to attend it. I couldn’t find an appointment at the Houston consulate for July, the entire month was booked but luckily someone cancelled last week so I made a quick trip to Houston. And I got my passport back today and saw the VISA was approved. So I’ll be attending FLOCK 2018 šŸ˜€

Added a search bar for the messages archive (https://pagure.io/fedora-commops/fedora-happiness-packets/issue/11). Djangos’ django.contrib.postgres.search module makes use of PostgreSQLā€™s full text search engine which made it pretty convenient to design a query to search against the messages, senders, and recipients. Using the SearchVector you can search against multiple fields. Maybe next week I can add an option to filter and sort the search results based on teh date or user.

Little bit of holdup on (https://pagure.io/fedora-infrastructure/issue/7053) I thought it would be straightforward setting up the production and staging instance once the dev instance was done but ran into a lot of bumps:

  1. Client IDs and secrets: In certain cases, you can’t always set the environment variables with the client ID and secret which is why I opted to go with a secret.json file for the staging and production instance.
  2. JWK endpoints: Mozillas’ OIDC Django plugin throws an error if the JWK doesn’t specify ‘alg’ (the algorithm for the key). I filed a bug report here (https://github.com/mozilla/mozilla-django-oidc/issues/247). The way I worked around it in the dev instace was using the public key from here (https://github.com/transtats/transtats/blob/devel/transtats/settings/auth.py) so I never had to use the endpoint. But I’m not entirely sure how to generate a public key for the production and staging instance from the JWK endpoint. So for now I’m thinking about making a PR to mozilla-django-oidc
  3. fedmsg: Finally, with the help of the infra team, got fedmsg to work in the staging instance. Had to publish messages to the relay.

Badge: So the badge is waiting on the logo for now (https://pagure.io/fedora-badges/issue/627). Might still be able to get this done before since fedmsg is up and running.

Design: The logo request (https://pagure.io/design/issue/606), blue doesn’t seem like the appropriate color to express happiness.

fedmsg on CentOS

Zeromq

fedmsg is “a library built on ZeroMQ using the PyZMQ Python bindings”. So I thought it might help to learn a little bit more about ZeroMQ ( which is not a messaging queue).

  • Contexts – You usually have one context per process and which manages the sockets.
  • Socket – It can be configured and connect to other sockets to exchange messages. You have different types for the different patterns. fedmsg uses the publish/subscribe pattern (asynchronous), where a single process sends messages to multiple recipients.
  • Channel – It’s the connection between two sockets.
  • Transport – fedmsg, in particular, uses the TCP protocol

The sockets used in fedmsg to publish and subscribe to messages isĀ  PUB (publisher) and SUB (subscriber). Messages, which are JSON encoded, are published with a topic which can be used by subscribers to filter the messages. A topic in fedmsg can take the form

<topic_prefix>.<environment>.<modname>.<topic>

modname is the module trying to publish the message

Setting it up

Disclaimer: This is for a development instance and not production.

Installing fedmsg is pretty straightforward but configuring it can be a bit tedious.

sudo yum install fedmsg

If you go to /etc/fedmsg.d/ you should see all these files: base.py, endpoints.py, gateway.py, ircbot.py, logging.py, relay.py, and ssl.py

If you try to publish a message from the interpreter

>> import fedmsg
>>> fedmsg.publish(topic='testing', modname='test', msg={
... 'test': "Hello World",
... })

You might get this error AttributeError: 'FedMsgContext' object has no attribute 'publisher'

(Refer https://github.com/fedora-infra/fedmsg/issues/426)

To fix this you need to configure the endpoints in endpoints.py

So an endpoint is basically a string which specifies a transport (TCP) and an address to bind to.

<transport>://address

You can read more about it here http://www.fedmsg.com/en/stable/configuration/#endpoints

In endpoints.py add your endpoint to the config dictionary. The key has to be the <name of the application> .<host name> and the value is a list of all possible endpoints the socket can bind to. You can find out your hostname using hostnameĀ -s

You can view the configuration from the terminal using fedmsg-config

One possible entry could look like this:

"__main__.centos-s-1vcpu": ["tcp://127.0.0.1:5001"],

If we try to publish the message now it should work. But how can I check if it shows up on the fedmsg bus? Launch another Python interpreter and try:

>> for name, endpoint, topic, msg in fedmsg.tail_messages():
... print topic, msg

This should print out all the messages currently arriving. You could also use fedmsg-tail –really-pretty from the terminal. fedmsg.tail_messages() gets messages published on the sockets listed in endpoints.

But if you try to publish now you might get this

IOError: Couldn't find an available endpoint for name '__main__.centos-s-1vcpu-1gb-lon1-01'

That’s because we specified one endpoint for this application in the config.py file. So if we change the config file to add another one.

"__main__.centos-s-1vcpu-1gb-lon1-01": ["tcp://127.0.0.1:5001", "tcp://127.0.0.1:5002"],

You should be able to publish messages now and see it

No handlers could be found for logger "fedmsg.crypto"
/usr/lib/python2.7/site-packages/fedmsg/core.py:441: UserWarning: !! invalid message received: {u'username': u'root', u'i': 3, u'timestamp': 1530124619, u'msg_id': u'2018-e4090d11-8944-4b30-9d25-0f59c549fab0', u'topic': u'org.fedoraproject.dev.test.testing', u'msg': {u'test': u'Hello World'}}
warnings.warn("!! invalid message received: %r" % e.msg)

You get this warning becauseĀ validate_signatures in ssl.py is enabled by default. You can disable it for now for development.

references

Fedora Happiness Packets on CentOS 7

I’m writing this for future me. My default OS is Ubuntu so setting up the project on CentOS was new for me.

gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong –param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong –param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c _scandir.c -o build/temp.linux-x86_64-2.7/_scandir.o
unable to execute gcc: No such file or directory
error: command ‘gcc’ failed with exit status 1

yum install gcc

gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong –param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong –param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c _scandir.c -o build/temp.linux-x86_64-2.7/_scandir.o
_scandir.c:14:20: fatal error: Python.h: No such file or directory
#include
^
compilation terminated.
error: command ‘gcc’ failed with exit status 1

yum install python-devel

gcc -pthread -shared -Wl,-z,relro build/temp.linux-x86_64-2.7/Modules/2.x/readline.o readline/libreadline.a readline/libhistory.a -L/usr/lib64 -lncurses -lpython2.7 -o build/lib.linux-x86_64-2.7/gnureadline.so
gcc: error: readline/libreadline.a: No such file or directory
gcc: error: readline/libhistory.a: No such file or directory
error: command ‘gcc’ failed with exit status 1

yum install patch

gcc -pthread -shared -Wl,-z,relro build/temp.linux-x86_64-2.7/Modules/2.x/readline.o readline/libreadline.a readline/libhistory.a -L/usr/lib64 -lncurses -lpython2.7 -o build/lib.linux-x86_64-2.7/gnureadline.so
/usr/bin/ld: cannot find -lncurses
collect2: error: ld returned 1 exit status
error: command ‘gcc’ failed with exit status 1

yum install ncurses-devel

GSoC update Week 1 and 2

Week 1:

Week 2 and 3:

OpenID Connect and Authenticating against Ipsilon

Step 1 – Get your credentials

You need to register your app with the OP https://iddev.fedorainfracloud.org/

I used https://github.com/puiterwijk/oidc-register for that.

pip install oidc-register

To register you need the provider, application, and redirect(callback) URLs

oidc-register https://iddev.fedorainfracloud.org/ https://127.0.0.1:8443 https://127.0.0.1:8443/oidc/callback/

Note: You can’t use http unless it’s with a localhost.Ā 

After you do that you should have a client_secrets.json file in the same directory with the client secret and ID

Step 2 – Update your settings

For https://github.com/mozilla/mozilla-django-oidc it’s pretty straightforward. Update or add the OIDC_RP_CLIENT_ID and OIDC_RP_CLIENT_SECRET in the settings.py file or better yet add it as an environment variable. You don’t want to accidentally commit the client ID and secret to your github repository.

export OIDC_RP_CLIENT_ID='xxx'
export OIDC_RP_CLIENT_SECRET='xxx'

Don’t forget to include the Authorizaation, Token and User Info endpoints in your settings file

You might also need to change the signing algorithm to RS256 which requires the public key of the OP or its JWK endpoint https://id.fedoraproject.org/openidc/Jwks

References

https://docs.pagure.org/infra-docs/dev-guide/auth.html

https://fedoraproject.org/wiki/Infrastructure/Authentication

 

© 2018 Anna Philips

Theme by Anders NorénUp ↑