Jekyll2023-11-29T17:42:19+00:00/feed.xmlThe log of Paul WilsonPaul Wilson's log on the web.Connect your Elixir Nerves devices to your Phoenix server over Websockets, with Fedecks2023-05-15T12:14:37+00:002023-05-15T12:14:37+00:00/elixir/phoenix/nerves/2023/05/15/websocket-connect-your-elixir-nerves-devices-to-your-phoenix-server-with-fedecks<blockquote>
<p>Fedecks makes it easy to establish duplex communication between your Elixir Nerves devices and your Phoenix Server in the cloud</p>
</blockquote>
<p>Whatever your <a href="https://nerves-project.org">Nerves</a> project does, there’s a good chance that it can be enhanced by securely connecting to a <a href="https://phoenixframework.org">Phoenix Server</a> in the cloud; it gives you the ability to can monitor and control your device from afar.</p>
<p>Websockets are a great medium for this connection. The provide two way communication without having to punch holes in your network router and they are secure (assuming your server is behind ssl). The can be a bit fiddly to set up though - except not any more. I’ve extracted a couple of hexicles from my projects to make things easier for you (and future me):</p>
<ul>
<li><em>Fedecks Server</em> is for the server side of things. Docs are <a href="https://hexdocs.pm/fedecks_server/readme.html">here</a> and the source is <a href="https://github.com/paulanthonywilson/fedecks_server">here</a>.</li>
<li><em>Fedecks Client</em> is for the Nerves side of things. Docs are <a href="https://hexdocs.pm/fedecks_client">here</a> and the source is <a href="https://github.com/paulanthonywilson/fedecks_client">here</a></li>
</ul>
<h2 id="fedecks-features">Fedecks Features</h2>
<ul>
<li><strong>Automatic re-establishes lost connection</strong>. If your home network is anything like mine, this is pretty handy. Combine with <a href="https://hexdocs.pm/vintage_heart/readme.html">VintageHeart</a> for extra network robustness.</li>
<li><strong>After initial authentication, connection is automatically established between reboots</strong>. Lose power or otherwise need to reboot your device? You don’t need to keep logging on.</li>
<li><strong>Handles pinging the server periodically to keep the connection awake</strong>. If your client does not do this then the server will close the connection; you’d automatically reconnect but you lose some connectivity.</li>
<li><strong>Closes and re-establishes the connection if <em>pong</em> not received for a <em>ping</em></strong>. Sometimes connections go down and the client receives no notifications, leaving a <em>zombie</em> that looks connected. Fedecks client takes care of that.</li>
<li><strong>No credentials are stored locally on the client</strong> Subsequent authentication is via a token signed by the server. The token is periodically refreshed so it should always be usable. Persisting the token enables the reconnection between reboots. The expiry is pretty long, which suits me, but you can <a href="https://hexdocs.pm/fedecks_server/FedecksServer.FedecksHandler.html">configure it to be shorter</a> on the server side.</li>
<li><strong>No need to decode / encode yourself or mess with JSON</strong>. Communicates with Elixir terms (but be aware that decoding is safe, so be careful or avoid using atoms).</li>
<li><strong>Token or credentials are sent in a header</strong>. I think it is simplest and safest to authenticate before upgrading the initial HTTP request to a websocket, but the initial request is a <code class="language-plaintext highlighter-rouge">GET</code>. Depending on your setup (eg proxied behind nginx), <code class="language-plaintext highlighter-rouge">GET</code> requests are prone to be logged to an access log which I think leaves credentials vulnerable to leaking if sent as request parameters. Headers seem like a safer option.</li>
</ul>
<p>I ran through the basics of <em>Fedecks</em> in my Lightning Talk at Elixir Conf EU 2023, but let’s run through a simple implementation.</p>
<h2 id="fedecks-server-setup">Fedecks Server setup</h2>
<p>In your Phoenix app, that you will eventually deploy to the cloud, the first and most obvious step is to add <code class="language-plaintext highlighter-rouge">fedecks_server</code> as a dependency.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="ss">:fedecks_server</span><span class="p">,</span> <span class="s2">"~> 0.1.2"</span><span class="p">}</span>
</code></pre></div></div>
<h3 id="fedecksserverfedeckshandler-implementation">FedecksServer.FedecksHandler implementation</h3>
<p>Then you will need to implement a <a href="https://hexdocs.pm/fedecks_server/FedecksServer.FedecksHandler.html">Fedecks Handler</a>.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyServerWeb</span><span class="o">.</span><span class="no">MySocketHandler</span> <span class="k">do</span>
<span class="nv">@behaviour</span> <span class="no">FedecksServer</span><span class="o">.</span><span class="no">FedecksHandler</span>
<span class="nv">@impl</span> <span class="no">FedecksHandler</span>
<span class="k">def</span> <span class="n">otp_app</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="ss">:my_server</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">authenticate?/1</code> callback also needs to be implemented. This will be called on initial authentication<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup> with credentials provided by the client in a map in whatever form you decide is appropriate. For example<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup></p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">def</span> <span class="n">authenticate?</span><span class="p">(%{</span>
<span class="s2">"username"</span> <span class="o">=></span> <span class="n">username</span><span class="p">,</span>
<span class="s2">"password"</span> <span class="o">=></span> <span class="n">password</span><span class="p">,</span>
<span class="s2">"fedecks-device-id"</span> <span class="o">=></span> <span class="n">_device_id</span>
<span class="p">})</span> <span class="k">do</span>
<span class="no">Plug</span><span class="o">.</span><span class="no">Crypto</span><span class="o">.</span><span class="n">secure_compare</span><span class="p">(</span><span class="n">username</span><span class="p">,</span> <span class="s2">"the_user"</span><span class="p">)</span> <span class="o">&&</span>
<span class="no">Plug</span><span class="o">.</span><span class="no">Crypto</span><span class="o">.</span><span class="n">secure_compare</span><span class="p">(</span><span class="n">password</span><span class="p">,</span> <span class="s2">"secure_password"</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Note that the framework will always add “fedecks-device-id” to the map. This could give you the opportunity to associate a device with a user. The device id is assumed to be a unique <code class="language-plaintext highlighter-rouge">String</code>.</p>
<p>Let’s also implement the <em>optional</em> <code class="language-plaintext highlighter-rouge">connection_established/1</code> callback, which will then get called every time the client establishes a connection, regardless of whether <code class="language-plaintext highlighter-rouge">authenticate?/1</code> was called. As this callback takes place in the socket’s process we can
subscribe to a topic that we can use to push messages to our Nerves device.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">def</span> <span class="n">connection_established</span><span class="p">(</span><span class="n">device_id</span><span class="p">)</span> <span class="k">do</span>
<span class="no">Phoenix</span><span class="o">.</span><span class="no">PubSub</span><span class="o">.</span><span class="n">subscribe</span><span class="p">(</span><span class="no">MyServer</span><span class="o">.</span><span class="no">PubSub</span><span class="p">,</span> <span class="s2">"nerves_downstream_topic.</span><span class="si">#{</span><span class="n">device_id</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div></div>
<p>We’ll also implement the optional <code class="language-plaintext highlighter-rouge">handle_info/2</code> to receive messages to send downstream.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">def</span> <span class="n">handle_info</span><span class="p">(</span><span class="n">_device_id</span><span class="p">,</span> <span class="p">{</span><span class="ss">:send_downstream</span><span class="p">,</span> <span class="n">message</span><span class="p">})</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:push</span><span class="p">,</span> <span class="n">message</span><span class="p">}</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Also let’s implement <code class="language-plaintext highlighter-rouge">handle_in/2</code> to handle any messages sent from the Nerves box.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">def</span> <span class="n">handle_in</span><span class="p">(</span><span class="n">device_id</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span> <span class="k">do</span>
<span class="no">Phoenix</span><span class="o">.</span><span class="no">PubSub</span><span class="o">.</span><span class="n">broadcast</span><span class="p">(</span>
<span class="no">MyServer</span><span class="o">.</span><span class="no">PubSub</span><span class="p">,</span>
<span class="s2">"nerves_upstream_topic.</span><span class="si">#{</span><span class="n">device_id</span><span class="si">}</span><span class="s2">"</span><span class="p">,</span>
<span class="p">{</span><span class="ss">:message_from_device</span><span class="p">,</span> <span class="n">message</span><span class="p">}</span>
<span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<h3 id="configuration">Configuration</h3>
<p>Now we will need to add some stuff to our config. Let’s assume <code class="language-plaintext highlighter-rouge">runtime.exs</code></p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="n">config</span> <span class="ss">:my_server</span><span class="p">,</span> <span class="no">MyServerWeb</span><span class="o">.</span><span class="no">MySocketHandler</span><span class="p">,</span>
<span class="ss">salt:</span> <span class="no">System</span><span class="o">.</span><span class="n">fetch_env!</span><span class="p">(</span><span class="s2">"FEDECKS_SALT"</span><span class="p">),</span>
<span class="ss">secret:</span> <span class="no">System</span><span class="o">.</span><span class="n">fetch_env!</span><span class="p">(</span><span class="s2">"FEDECKS_SECRET"</span><span class="p">)</span>
</code></pre></div></div>
<p>The salt and secret can both be generated with <code class="language-plaintext highlighter-rouge">mix phx.gen.secret</code> and are used to sign the token used for re-authentication.</p>
<p>See <a href="https://hexdocs.pm/fedecks_server/FedecksServer.FedecksHandler.html">the handler documentation</a> for other optional callbacks and configuration options.</p>
<h3 id="add-to-the-endpoint">Add to the EndPoint</h3>
<p><a href="https://hexdocs.pm/fedecks_server/FedecksServer.Socket.html#fedecks_socket/2">`FedecksServer.Socket.fedecks_socket/2</a> is a handy macro for adding Fedecks to your endpoint.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="k">defmodule</span> <span class="no">MyServerWeb</span><span class="o">.</span><span class="no">Endpoint</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Phoenix</span><span class="o">.</span><span class="no">Endpoint</span><span class="p">,</span> <span class="ss">otp_app:</span> <span class="ss">:my_server</span>
<span class="kn">import</span> <span class="no">FedecksServer</span><span class="o">.</span><span class="no">Socket</span><span class="p">,</span> <span class="ss">only:</span> <span class="p">[</span><span class="ss">fedecks_socket:</span> <span class="mi">1</span><span class="p">]</span>
<span class="c1">## etc...</span>
<span class="n">fedecks_socket</span><span class="p">(</span><span class="no">MyServerWeb</span><span class="o">.</span><span class="no">MySocketHandler</span><span class="p">)</span>
<span class="c1"># etc ...</span>
<span class="k">end</span>
</code></pre></div></div>
<p>As we’ve omitted the path it defaults to mounting the socket at “/fedecks”, or actually “fedecks/websocket” as Phoenix needs it to be differentiated from a long polling request, see <a href="https://hexdocs.pm/phoenix/Phoenix.Socket.Transport.html">`Phoenix.Socket.Transport’</a>.</p>
<h2 id="fedecks-client-setup">Fedecks Client Setup</h2>
<p>Now let’s set up the Nerves client. Assuming you have installed Nerves, you can <a href="https://hexdocs.pm/nerves/getting-started.html#creating-a-new-nerves-app">create a new Nerves project</a>. (Alternatively just create a new Elixir mix project, with a supervision tree, for trying things out locally.)</p>
<p>Once that’s done add fedecks client as dependency.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="p">{</span><span class="ss">:fedecks_client</span><span class="p">,</span> <span class="s2">"~> 0.1"</span><span class="p">}</span>
</code></pre></div></div>
<h3 id="create-a-client-module">Create a Client module</h3>
<p>Now let’s create the client.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyNerves</span><span class="o">.</span><span class="no">SocketClient</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">FedecksClient</span>
</code></pre></div></div>
<p>Now we have to implement <code class="language-plaintext highlighter-rouge">device_id/0</code> to provide a unique id. For our purposes the hostname provided by Nerves should be enough<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote" rel="footnote">3</a></sup>.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">def</span> <span class="n">device_id</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">hostname</span><span class="p">}</span> <span class="o">=</span> <span class="ss">:inet</span><span class="o">.</span><span class="n">gethostname</span><span class="p">()</span>
<span class="n">to_string</span><span class="p">(</span><span class="n">hostname</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div></div>
<p>We also need to provide the connection url. Chances are that you will want to load this from config, and vary it by <code class="language-plaintext highlighter-rouge">Mix.target/0</code> and/or <code class="language-plaintext highlighter-rouge">Mix.env/0</code>. For now we’ll run things on our development machines and point to localhost<sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote" rel="footnote">4</a></sup>.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">def</span> <span class="n">connection_url</span> <span class="k">do</span>
<span class="s2">"ws://localhost:4000/fedecks/websocket"</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Note that for secure connections we will want the protocol to be “wss://”</p>
<p>See <a href="https://hexdocs.pm/fedecks_client/FedecksClient.html#callbacks">the documentation</a> for other optional callbacks to implement.</p>
<h3 id="add-to-the-application-supervision-tree">Add to the application supervision tree</h3>
<p>In your <code class="language-plaintext highlighter-rouge">application.ex</code></p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">def</span> <span class="n">start</span><span class="p">(</span><span class="n">_type</span><span class="p">,</span> <span class="n">_args</span><span class="p">)</span> <span class="k">do</span>
<span class="n">opts</span> <span class="o">=</span> <span class="p">[</span><span class="ss">strategy:</span> <span class="ss">:one_for_one</span><span class="p">,</span> <span class="ss">name:</span> <span class="no">MyNerves</span><span class="o">.</span><span class="no">Supervisor</span><span class="p">]</span>
<span class="n">children</span> <span class="o">=</span> <span class="p">[</span><span class="no">MyNerves</span><span class="o">.</span><span class="no">SocketClient</span><span class="p">]</span> <span class="o">++</span> <span class="n">children</span><span class="p">(</span><span class="n">target</span><span class="p">())</span>
<span class="no">Supervisor</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="n">children</span><span class="p">,</span> <span class="n">opts</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div></div>
<h2 id="try-fedecks-out">Try Fedecks out</h2>
<p>Now we can try things out. From your Nerves directory run <code class="language-plaintext highlighter-rouge">iex -S mix</code>. If you run <code class="language-plaintext highlighter-rouge">:observer.start</code> then you should see the Fedecks Client processes in your supervision tree. Subscribe to get events and to check your “device id”.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="o">></span> <span class="ss">:observer</span><span class="o">.</span><span class="n">start</span>
<span class="ss">:ok</span>
<span class="n">iex</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span><span class="o">></span> <span class="no">MyNerves</span><span class="o">.</span><span class="no">SocketClient</span><span class="o">.</span><span class="n">subscribe</span><span class="p">()</span>
<span class="ss">:ok</span>
<span class="n">iex</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span><span class="o">></span> <span class="no">MyNerves</span><span class="o">.</span><span class="no">SocketClient</span><span class="o">.</span><span class="n">device_id</span><span class="p">()</span>
<span class="s2">"Ossian"</span>
</code></pre></div></div>
<p>“Ossian” is the name of my development laptop for reasons that I can not rightly remember. On a Nerves devices it will be something like “nerves-242a”.</p>
<p>From your Phoenix Server directory <code class="language-plaintext highlighter-rouge">iex -S mix phx.server</code>. Let’s subscribe to receive messages from the client.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span><span class="o">></span> <span class="no">Phoenix</span><span class="o">.</span><span class="no">PubSub</span><span class="o">.</span><span class="n">subscribe</span><span class="p">(</span><span class="no">MyServer</span><span class="o">.</span><span class="no">PubSub</span><span class="p">,</span> <span class="s2">"nerves_upstream_topic.Ossian"</span><span class="p">)</span>
<span class="ss">:ok</span>
</code></pre></div></div>
<p>Obviously substitute “Ossian” above with your development machine’s hostname.</p>
<p>Ok, let’s try stuff out.</p>
<p>From your <strong>client</strong> iex console</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span><span class="o">></span> <span class="no">MyNerves</span><span class="o">.</span><span class="no">SocketClient</span><span class="o">.</span><span class="n">login</span><span class="p">(%{</span><span class="s2">"username"</span> <span class="o">=></span> <span class="s2">"the_user"</span><span class="p">,</span> <span class="s2">"password"</span> <span class="o">=></span> <span class="s2">"secure_password"</span><span class="p">})</span>
<span class="ss">:ok</span>
<span class="n">iex</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span><span class="o">></span> <span class="n">flush</span>
<span class="p">{</span><span class="no">MyNerves</span><span class="o">.</span><span class="no">SocketClient</span><span class="p">,</span> <span class="ss">:connecting</span><span class="p">}</span>
<span class="p">{</span><span class="no">MyNerves</span><span class="o">.</span><span class="no">SocketClient</span><span class="p">,</span> <span class="ss">:connected</span><span class="p">}</span>
</code></pre></div></div>
<p>Now let’s send a message from the <strong>client</strong> to the <strong>server</strong></p>
<p><strong>On the client</strong></p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="p">(</span><span class="mi">6</span><span class="p">)</span><span class="o">></span> <span class="no">MyNerves</span><span class="o">.</span><span class="no">SocketClient</span><span class="o">.</span><span class="n">send</span><span class="p">({</span><span class="s2">"hello"</span><span class="p">,</span> <span class="s2">"matey"</span><span class="p">})</span>
<span class="ss">:ok</span>
</code></pre></div></div>
<p><strong>On the server</strong></p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span><span class="o">></span> <span class="n">flush</span>
<span class="p">{</span><span class="ss">:message_from_device</span><span class="p">,</span> <span class="p">{</span><span class="s2">"hello"</span><span class="p">,</span> <span class="s2">"matey"</span><span class="p">}}</span>
<span class="ss">:ok</span>
</code></pre></div></div>
<p>Now send a message from the <strong>server</strong> to the <strong>client</strong>.</p>
<p><strong>on the server</strong>, substituting “Ossian” for you computer’s hostname.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span><span class="o">></span> <span class="no">Phoenix</span><span class="o">.</span><span class="no">PubSub</span><span class="o">.</span><span class="n">broadcast</span><span class="p">(</span><span class="no">MyServer</span><span class="o">.</span><span class="no">PubSub</span><span class="p">,</span> <span class="s2">"nerves_downstream_topic.Ossian"</span><span class="p">,</span> <span class="p">{</span><span class="ss">:send_downstream</span><span class="p">,</span> <span class="p">[</span><span class="s2">"hi"</span><span class="p">,</span> <span class="s2">"there"</span><span class="p">]})</span>
<span class="ss">:ok</span>
</code></pre></div></div>
<p>Now <strong>on the client</strong></p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="p">(</span><span class="mi">7</span><span class="p">)</span><span class="o">></span> <span class="n">flush</span>
<span class="p">{</span><span class="no">MyNerves</span><span class="o">.</span><span class="no">SocketClient</span><span class="p">,</span> <span class="p">{</span><span class="ss">:message</span><span class="p">,</span> <span class="p">[</span><span class="s2">"hi"</span><span class="p">,</span> <span class="s2">"there"</span><span class="p">]}}</span>
<span class="ss">:ok</span>
</code></pre></div></div>
<p>There we go: two way arbitrary communication between the client and the server. There’s one caveat: terms are decoded safely on either end, so unknown atoms will not decode.</p>
<h2 id="future-developments">Future developments?</h2>
<p>After the Lightning Talk at Elixir Conf in Lisbon, <a href="https://mat.geeky.net">Mat Trudel</a> told me about the work he had done to Phoenix to make Websockets much more flexible with <a href="https://hexdocs.pm/websock">WebSock</a>. I would like to support that approach to upgrading to Websockets in plugs and/or controllers in Phoenix 1.7 +.</p>
<p>I may also consider rolling presence tracking and message passing over Phoenix PubSub from my personal projects up into <em>Fedecks Server</em>.</p>
<hr />
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>Assuming the device is not offline for a long period then you only need to authenticate this way one time: subsequent authentication is done with a signed token provided (and frequently refreshed) by the server. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>Let’s take it as read that we’re not really going to be hardcoding usernames and passwords. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>It’s good enough for my projects, though <a href="https://github.com/nerves-project/boardid#caveats">not absolutely guaranteed unique</a> <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>Obviously this is just for trying things out. Remember though that if you do want to try from firmware on a device to your development machine (host), by default Nerves does not come with a ZeroConf MDN client so local computer names will not work; you will need to point it to your development box’s IP address. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Paul WilsonFedecks makes it easy to establish duplex communication between your Elixir Nerves devices and your Phoenix Server in the cloudElixirConf EU 20232023-05-01T04:24:26+00:002023-05-01T04:24:26+00:00/elixir/2023/05/01/elixirconf-eu-2023<p>Conference experience reports used to be a thing, back in the days of people doing lots of blogging. Why not be a bit retro, and put one together?</p>
<p>This year’s Elixir Conf in Lisbon was the first conference I’ve attended in-person since Prague in 2019. For various reasons, including the negotiations for the <a href="https://techcrunch.com/2019/08/01/amazon-backed-food-delivery-startup-deliveroo-acquires-edinburgh-software-studio-cultivate/">Deliveroo acquisition of Cultivate</a><sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>, I spent a lot of the 2019 conference preparing the slides <a href="https://www.softwaretalks.io/v/8842/paul-wilson-a-production-grade-nerves-system-elixirconf-eu-2019">for my talk</a>. This time I was an attendee so could relax and take in the other talks.</p>
<p>This is a brief summary of what I saw and from what I can decipher of my notes. It was a 2.5 track conference; I did not attend every talk.</p>
<h2 id="josé-valim-keynote">José Valim keynote</h2>
<p>Elixir 1.15 is to be the most boring Elixir version release so far, but in a good way. There are to be developer-experience improvements for large codebases, consisting of up to 40% faster compilations when used with (the upcoming) OTP 26. There’ll also be improvements in OTP 26 for the experience of developing on Windows. One of the speed improvements for maps in OTP 26 could break anyone relying on key order when iterating maps (which no-one should be doing as it has never been formally guaranteed).</p>
<p>He also mentioned the work on bringing “Set Theoretic Types” to Elixir, which is progressing very well and a the documentation / feedback stage but mostly pointing to the later talk by Guillaume Duboc. Development will start later this year.</p>
<p>Improvements to the developer experience include <a href="https://hexdocs.pm/mix/Mix.html#install/2">Mix.install</a>, <a href="https://hexdocs.pm/elixir/Kernel.html#dbg/2">dbg</a><sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup>, <a href="https://hexdocs.pm/mix/main/Mix.Tasks.Format.html#module-plugins">plugins for <code class="language-plaintext highlighter-rouge">mix format</code></a>. A promising area is the introduction of <em>code fragments</em> which enable tools (eg <a href="https://github.com/elixir-lsp/elixir-ls">Elixir LS</a>) can work better with code that can not currently compile.</p>
<p>The rest of the keynote was some demonstrations of cool LiveBook features, such as <a href="https://hexdocs.pm/kino/Kino.Process.html#render_seq_trace/2">Kino.Process.render_seq_trace/2</a> for visualising function calls with a sequence diagram, or <a href="https://hexdocs.pm/kino/Kino.Process.html#render_sup_tree/2">Kino.Process.render_sup_tree/2</a> to show a supervision tree.</p>
<p>José also demonstrated some of the new LiveBook machine learning capabilities in the <a href="https://news.livebook.dev/build-and-deploy-a-whisper-chat-app-to-hugging-face-in-15-minutes---launch-week-1---day-4-wYM0w">Smart Cells</a> and also how Smart Cells can be used for extension by converting to code then changing.</p>
<p>There was a great answer to an audience question: why make LiveBook when <a href="https://jupyter.org">Jupyter notebooks</a> already exists? Partly it was a proof of the the capabilities of LiveView. It was also that Jupyter uses global state across all users, when LiveBooks are designed to be shared but have individual state for each user/session. There are other advantages of being focussed on a single functional language, such as Smart Cells and the ability to detect whether an update makes another cell stale.</p>
<h2 id="a-domain-specific-language-for-impact--simon-de-haan-and-federico-meini">A Domain Specific Language for Impact — Simon de Haan and Federico Meini</h2>
<p><a href="https://www.turn.io">Turn.io</a> is a chat product aimed to social impact. The principal is that WhatsApp is available and used widely especially in developing countries. The automatic chat is configured with a powerful External (I think) Domain Specific Language (DSL).</p>
<p>The cool part is that the DSL is similar enough to Elixir that they do not need a parser. <a href="https://hexdocs.pm/elixir/Code.html#string_to_quoted/2"><code class="language-plaintext highlighter-rouge">Code.string_to_quoted/2</code></a> will turn the user’s DSL to AST which can then be transformed by a bit of code into Structs that can be saved to the database. Even cooler they can go the other way by turning their Structs to AST then running <a href="https://hexdocs.pm/elixir/Macro.html#to_string/1"><code class="language-plaintext highlighter-rouge">Macro.to_string/1</code></a>.</p>
<p>As well as the DSL they have a visual way of building the automation flow. For that they convert the visual representation to JSON that conforms to the <a href="https://flowinterop.org">Flow Interoperability Standard</a> which is a JSON representation of a state machine.</p>
<p>They do need to do their own parsing for part of the flow. That is for guard expressions such as making a decision base on the age of the contact eg “contact.patient_age > 18”. For this they use <a href="https://hexdocs.pm/nimble_parsec/NimbleParsec.html">Nimble Parsec</a>. There was a comment disguised as a comment, suggesting that the Abacus maths parsing hexicle should be used rather doing their own parsing. The <a href="https://github.com/narrowtux/abacus">Abacus Github README</a> overview is not included in its <a href="https://hexdocs.pm/abacus/api-reference.html">Hex documentation</a>.</p>
<h2 id="remote-debugging-with-livebook--luca-dei-zotti">Remote Debugging with Livebook — Luca Dei Zotti</h2>
<p>This was a nice little demo of debugging an issue on a “production” instance with tools like <a href="https://www.erlang.org/doc/man/dbg.html">dbg</a> with <a href="https://www.erlang.org/doc/man/dbg.html#trace_port-2">trace port</a> remotely from a LiveBook rather than a remote console. Also included some scary code replacement by compiling on the server, without deploying; I am unsure about this part but “needs must as the devil drives” I guess.</p>
<p>I can see how LiveBook might beat a remote console for this, in that there is a clear path of what has already been investigated.</p>
<h2 id="optimising-liveview-for-realtime-applications--marius-saraiva">Optimising LiveView for Realtime Applications — Marius Saraiva</h2>
<p>Marius is the person behind <a href="https://hexdocs.pm/surface/Surface.html">Surface</a>, which I haven’t used, and his examples used <code class="language-plaintext highlighter-rouge">Surface</code> and the <a href="https://hexdocs.pm/surface/Surface.html#sigil_F/2"><code class="language-plaintext highlighter-rouge">~F</code></a> sigil which confused and distracted me a little until I realised. I am dumb.</p>
<p>I loved all the tips in this talk. While Marius started by saying that not all Live View needs to be optimised, most of the techniques he demonstrated now seem more like good practice rather than premature optimisation.</p>
<p>Marius started by reminding us of how LiveView, in particular <a href="https://hexdocs.pm/phoenix_live_view/Phoenix.Component.html#sigil_H/2">HEEx</a> works: splitting the dynamic and static parts of a template and after the initial rendering only sending the dynamic parts that have changed to the client. The tips were all about minimising those changes.</p>
<h3 id="replace-function-calls-with-components">Replace function calls with components</h3>
<p>In the olden times of a few years ago, it was common to call functions from within templates to return strings to display.</p>
<p>eg something like</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><div></span>
<span class="nt"><</span><span class="err">%=</span> <span class="na">score</span><span class="err">(@</span><span class="na">score</span><span class="err">)</span> <span class="err">%</span><span class="nt">></span>
<span class="nt"></div></span>
</code></pre></div></div>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defp</span> <span class="n">score</span><span class="p">(</span><span class="n">score</span><span class="p">)</span> <span class="k">do</span>
<span class="s2">"You have </span><span class="si">#{</span><span class="nv">@score</span><span class="si">}</span><span class="s2"> points"</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Then all the text will be sent every time the score changes. If this is replaced with a component function then only the score will
be sent on a change.</p>
<pre><code class="language-ruby_html"><div>
<.score score={@score} />
</div>
</code></pre>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defp</span> <span class="n">score</span><span class="p">(</span><span class="n">assigns</span><span class="p">)</span> <span class="k">do</span>
<span class="sx">~H""</span><span class="s2">"
You have </span><span class="si">#{</span><span class="nv">@score</span><span class="si">}</span><span class="s2"> points
"""</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Generally calling out to functions from a template is now a red flag.</p>
<h3 id="use-css-selectors-instead-of-dynamically-creating-classes">Use CSS selectors instead of dynamically creating classes</h3>
<p>Particularly now that Tailwind has become the default CSS framework for Phoenix and Live View, it is common to dynamically assign classes to reflect some changing state such as a button being disabled. Rather then send the large list of classes down the wire, this can be done purely in CSS by selecting on the disabled state….</p>
<p>eg</p>
<pre><code class="language-ruby_html"><button class="mybtn" disabled={@disabled}>Do it!</button>
</code></pre>
<p>in CSS</p>
<div class="language-css highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">.mybtn</span> <span class="p">{</span>
<span class="err">@</span><span class="py">apply</span><span class="p">:</span> <span class="n">bg-green-500</span> <span class="n">text-white</span> <span class="n">font-bold</span> <span class="n">py-3</span> <span class="n">px-4</span> <span class="n">rounded-lg</span> <span class="n">text-center</span>
<span class="p">.</span><span class="n">mybtn</span><span class="p">[</span><span class="n">disabled</span><span class="p">]</span> <span class="err">{</span>
<span class="err">@</span><span class="n">apply</span><span class="p">:</span> <span class="n">bg-gray-500</span> <span class="n">text-white</span> <span class="n">opacity-50</span> <span class="n">font-bold</span> <span class="n">py-3</span> <span class="n">px-4</span> <span class="n">rounded-lg</span> <span class="n">text-center</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Data and aria attributes can be used for more general selecting. I love this tip which is as cunning as a fox who’s just been appointed Professor of Cunning at Oxford University<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote" rel="footnote">3</a></sup>.</p>
<h3 id="avoid-accessing-whole-objects-when-parts-of-the-object-are-frequently-updated">Avoid accessing whole objects when parts of the object are frequently updated</h3>
<p>Marius demonstrated that when passing an <em>object</em> (ie map / struct) to a function component then the whole component is re-rendered and with all the data being passed to client when any part changes. The example was a <em>full name</em> component which was re-rendered whenever the same map’s <code class="language-plaintext highlighter-rouge">updated_at</code> value (displayed elsewhere) changed. Something like</p>
<pre><code class="language-ruby_html"><.full_name user={@user}/>
</code></pre>
<p>I am unable to reproduce this with vanilla Phoenix (1.7.1) LiveView (0.18.6). I could be getting the issue wrong but I do wonder if this is something to do with <em>Surface</em>.</p>
<h3 id="debounce--batch-very-frequent-updates">Debounce / Batch very frequent updates</h3>
<p>Last Marius demonstrated displaying simulated messages to a user that come at a ridiculous rate. Over a throttled connection downstream messages to the user become queued up, probably leading to eventual resource issues on the server. Batching the messages prevents this backlog and looks at least as well as sending individual messages.</p>
<p>I have probably spent too long on this talk. It was clear, actionable, with great examples.</p>
<h2 id="quantum-doodle-digital-twins-for-everyday-activities--paul-valckenaers">Quantum Doodle: Digital Twins for Everyday Activities — Paul Valckenaers</h2>
<p>I was intrigued by this talk and I am still intrigued. My understanding is that a Digital Twin is highly granular representation of a real world object, such as an aeroplane engine, that is continuously updated with information from sensors etc… It is used for things like monitoring and diagnosis of issues.</p>
<p>Paul was suggesting introducing digital twins for activities, such as student examinations. Honestly he lost me a lot. I left confused about what the difference would be between a Digital Twin for (say) items being shipped around the world and how that would be currently modelled in existing software.</p>
<h2 id="powerful-machine-learning-at-your-fingertips--jonatan-kłosko">Powerful Machine Learning at Your Fingertips — Jonatan Kłosko</h2>
<p>This was a great introduction of machine learning capabilities in Elixir. Jonatan gave an introduction to pre-trained models from <a href="https://huggingface.co">Hugging Face</a> and <a href="https://hexdocs.pm/bumblebee/Bumblebee.html">Bumblebee</a>.</p>
<p>Livebook featured heavily in this demonstration including using <a href="https://news.livebook.dev/v0.6-automate-and-learn-with-smart-cells-mxJJe">Smart Cells</a> to demonstrate and generate code to explore.</p>
<p>Also featured was running an AI chat in a distributed cluster with LiveView on <a href="https://huggingface.co/docs/hub/spaces">Hugging Face Spaces</a>.</p>
<h2 id="lively-liveview-with-membrane--lars-wikman">Lively LiveView with Membrane — Lars Wikman</h2>
<p>While watching Lars’ presentation I first became aware that it was automatically transcribing his words, then that when he said certain words it would load the next slide. He later showed that it was coded in LiveView, and using <a href="https://dockyard.com/blog/2023/03/07/audio-speech-recognition-in-elixir-with-whisper-bumblebee">Whisper with Bumblebee</a>. It also used <a href="https://hexdocs.pm/membrane">Membrane</a> for (I think) creating a waveform in SVG from the audio and <a href="https://hexdocs.pm/evision">Evision</a> for facial recognition.</p>
<p>The talk itself was largely philosophical and posed a question:</p>
<blockquote>
<p>Are things hard because cool things are hard, or are hard things cool?</p>
</blockquote>
<p>The answer is probably yes.</p>
<p>Lars wrote his conference experience up <a href="https://underjord.io/elixirconf-eu-2023-lisbon.html">here</a>.</p>
<h2 id="bringing-types-to-elixir--guillaume-duboc">Bringing Types to Elixir — Guillaume Duboc</h2>
<p>This was on at the same time as the <em>Lively LiveViw with Membrane</em> talk, but I cheated and caught up on a rewound live-stream.</p>
<p>Guillame walked through the <em>Set Theoretic Types</em> coming to Elixir. The concept of set types fits with how functions can work with pattern matching: if a function can take, say, <code class="language-plaintext highlighter-rouge">User.t()</code> or an <code class="language-plaintext highlighter-rouge">integer()</code> (denoting an id) then that is the set of acceptable input types.</p>
<p>The <em>Set Theoretic Types</em> (I’ll use STT from now on) seem to build on what is available already with <a href="https://hexdocs.pm/elixir/typespecs.html">Typespecs</a> but are more capable and are planned to be more integrated with the compiler: an end to digging through lots of code to figure out why dialyzer is complaining that function “has no local return”.</p>
<p>One interesting enhancement is <em>interesection</em> types. An example was a function <code class="language-plaintext highlighter-rouge">negate/1</code> that returns the negative of an integer and the opposite of a boolean. Rather than specifying this as a <em>union</em> type</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">@spec</span> <span class="n">negate</span><span class="p">(</span><span class="n">integer</span><span class="p">()</span> <span class="o">|</span> <span class="n">boolean</span><span class="p">())</span> <span class="p">::</span> <span class="n">integer</span><span class="p">()</span> <span class="o">|</span> <span class="n">boolean</span><span class="p">()</span>
</code></pre></div></div>
<p>with STT, we could be more specific</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$(integer() -> integer()) and (boolean() -> boolean())
</code></pre></div></div>
<p>Whether it’s good practice to vary the return type depending on the input type is another matter.</p>
<p>If I understand things, other features will include</p>
<ul>
<li><a href="https://hexdocs.pm/elixir/Kernel.html#guards">guard clauses</a> being integrated with the type system, showing more warnings (and errors) at compile time<sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote" rel="footnote">4</a></sup></li>
<li>Composing protocols, eg <code class="language-plaintext highlighter-rouge">$ type traversable(a) = Enumerable.t(a) and Collectable.t()</code></li>
<li>Parametric polymorphism — ie being able to define a parameter as an Enumerable of integers <code class="language-plaintext highlighter-rouge">Enumerable.t(integer())</code></li>
<li>Support for dynamic types, so the codebases will not have to go strongly typed all at once.</li>
</ul>
<p>There was no mention of how types will work with message passing between processes. I guess that is another reason for dynamic types.</p>
<h2 id="lightning-talks">Lightning talks</h2>
<p>I gave one of the 5 minute talks at the end of the day. I am glad I submitted slides rather than tried to do something with LiveBook, as I was considering — swapping laptops for those that needed was (as is usual) problematic.</p>
<h3 id="erlang-ecosystem-foundation---community-engagement--alistair-woodman-and-francesco-cesarini">Erlang Ecosystem Foundation - community engagement — Alistair Woodman and Francesco Cesarini</h3>
<p>This was a compelling pitch for joining the <a href="https://erlef.org">EEF</a>. I’ve just signed up.</p>
<h3 id="optimizing-software-delivery-on-the-fly--tom-calloway">Optimizing Software Delivery on-the-fly — Tom Calloway</h3>
<p>A super pitch for Tom’s product, <a href="https://www.kanbran.com/express-interest-in-beta-program-3a475c53">Kanbran</a> which is “like Kanban but with more fibre”.</p>
<h3 id="hiring-elixir-devs-in-2023--arjun-gillard">Hiring Elixir Devs in 2023 — Arjun Gillard</h3>
<p>A talk by Arjun Gillard, a recruiter, on how to hire and retain Elixir Developers. Worth watching when the videos come out, if that’s what you need to do.</p>
<p>PS - you could maybe also hire me.</p>
<h3 id="introducing-fedecks-for-easy-communication-between-nerves-and-your-phoenix-server--me">Introducing Fedecks, for easy communication between Nerves and your Phoenix Server — ME</h3>
<p>I talked about a set of libraries<sup id="fnref:5" role="doc-noteref"><a href="#fn:5" class="footnote" rel="footnote">5</a></sup> I have just released to make it very easy to set up a durable Websocket connection between a <a href="https://nerves-project.org">Nerves</a> device and a Phoenix Server living in the cloud.</p>
<p>They are <a href="https://hexdocs.pm/fedecks_server">Fedecks Server</a> and <a href="https://hexdocs.pm/fedecks_client">Fedecks Client</a>. I’ll make a post launching this a few days from posting this.</p>
<h3 id="liveview-goes-k6--sebastian-göttschkes">LiveView goes k6 — Sebastian Göttschkes</h3>
<p>About the <a href="https://k6.io">pK6 framework</a> for load testing from Grafana and how it integrates with Phoenix and LiveView.</p>
<h3 id="quick-tricks-with-the-iexexs-file--daniils-petrovs">Quick Tricks with the .iex.exs file — Daniils Petrovs</h3>
<p>Enhancing your <code class="language-plaintext highlighter-rouge">iex</code> experience by loading lots of things from your <code class="language-plaintext highlighter-rouge">.iex.exs</code> file.</p>
<p>https://gist.github.com/DaniruKun/ccbaad8720c203fd6d86a39722c63c51</p>
<h3 id="doomguy-visits-the-beam--andré-albuquerque">Doomguy visits the BEAM — André Albuquerque</h3>
<p>This was an amazing demonstration of supervision strategies with the Doom video game. Using what was clearly magic, loads an instance of Doom with monsters spawned by supervisors. <code class="language-plaintext highlighter-rouge">:one_for_one</code>, <code class="language-plaintext highlighter-rouge">:rest_for_one</code>, and <code class="language-plaintext highlighter-rouge">:one_for_all</code> was clearly demonstrated by <em>killing the monsters with a gun and observing the respawning</em>. Dynamic supervisors involved audience members joining via LiveView but didn’t work quite as expected as <code class="language-plaintext highlighter-rouge">ngrok</code> failed.</p>
<p>The code is <a href="https://github.com/amalbuquerque/doom-supervisor">here</a>.</p>
<h3 id="an-introduction-to-property-based-testing--roland-tritsch">An Introduction to Property Based Testing — Roland Tritsch</h3>
<p>Digital nomad, Roland, gie a quick overview of property testing.</p>
<h3 id="improving-democracy-with-a-petitions-platform-with-delegation--hector-perez-arenas">Improving democracy with a petitions platform with delegation — Hector Perez Arenas</h3>
<p>Hector showed us his <a href="https://youcongress.org">You Congress</a>, a better petitions site with the ability to downvote as well as upvote, and the ability to delegate your votes to someone trusted.</p>
<h2 id="lessons-from-using-elixir-and-phoenix-to-build-a-city-software-infrastructure--shankar-dhanasekaren">Lessons from using Elixir and Phoenix to build a city software infrastructure — Shankar Dhanasekaren</h2>
<p>This was an unusual, but fascinating, keynote to kick off day 2. Elixir Conf keynotes are, as far as I remember, heavily technology focussed. Shankar is CTO of Auroville. I had not heard of the city before, but Wikipedia <a href="https://en.wikipedia.org/wiki/Auroville">tells us</a> that it is an <em>experimental township</em> in India, founded by Mirra Alfassa, a spiritual guru.</p>
<p>Previously much of the software was developed with <a href="https://www.drupal.org">Drupal</a> but that has (is being?) replaced with Elixir and Phoenix which fit the city’s constraints:</p>
<ul>
<li>They only have a small team of developers</li>
<li>They need the flexibility to release, and then add more features later<sup id="fnref:6" role="doc-noteref"><a href="#fn:6" class="footnote" rel="footnote">6</a></sup>.</li>
<li>Overheads have to be low</li>
<li>The user interface needs to be rich</li>
<li>The budget is limited</li>
</ul>
<p>Scale is not an issue but Elixir + Phoenix (with LiveView) is a great fit as it is so productive.</p>
<p>Something notable, but not mentioned, was that they write all their own software rather than use services. For instance one of the many products is a booking system for their guest house. I would have liked to have known whether buying or renting in a commercial system was considered and what was the decision making process. I did not raise my hand to ask as I could not work out how to phrase the question without it coming across as a criticism.</p>
<p>It was also interesting to see that they are experimenting with growing their own food with <a href="https://farm.bot">Farm Bots</a>, which I guess are a commercial product.</p>
<h2 id="safer-db-migrations-with-excellent_migrations--artur-sulej">Safer DB Migrations with excellent_migrations — Artur Sulej</h2>
<p><a href="https://hexdocs.pm/excellent_migrations/readme.html">Excellent Migrations</a> is a library to detect potentially problematic issues with a migration, such as setting an existing column to be “not null”. These are the issues that may not appear when testing the migration but blow up in production with heavily populated tables. Nice points are that:</p>
<ul>
<li>It is implemented as a Credo check making it easy to add to what is likely to be an existing workflow</li>
<li>It is easy to configure, such as ignoring all migrations before a certain date as they’ve already gone to production or removing particular checks which may no longer be an issue with the database / version being used.</li>
<li>Checks can be ignored by adding a structured comment, if for example we know that the table in question is small.</li>
</ul>
<p>Artur gave a nice run through of some of how it is implemented by pattern matching on the AST.</p>
<h2 id="phoenix-beyond-cowboy--mat-trudel">Phoenix beyond Cowboy — Mat Trudel</h2>
<p><a href="https://hexdocs.pm/bandit/Bandit.html">Bandit</a> is an alternative to using <a href="https://github.com/ninenines/cowboy">Cowboy</a> with Phoenix. As I’ve felt a bit iffy about Cowboy since I found out <a href="https://erlang.org/pipermail/erlang-questions/2018-February/094835.html">the origin of its name</a>, I will definitely be trying it out. Other good reasons are that it is written in Elixir using OTP, and Mat showed Bandit coming out ahead in benchmarks that he’d run. I have found Cowboy’s code hard to follow, partly because it’s Erlang but also as it does its own process lifecycle management rather than using GenServers<sup id="fnref:7" role="doc-noteref"><a href="#fn:7" class="footnote" rel="footnote">7</a></sup>.</p>
<p>Prior to 1.7, Phoenix was tied to Cowboy because of the way websockets were implemented. While standard requests were built atop <a href="https://hexdocs.pm/plug/readme.html">Plug</a> making it possible to swap out the underlying server, websocket support was built directly using Cowboy’s <a href="https://ninenines.eu/docs/en/cowboy/2.10/manual/cowboy_websocket/">Websocket handler</a>. Now, thanks to Mat, Phoenix uses <a href="https://github.com/phoenixframework/websock">Websock</a> and <a href="https://github.com/phoenixframework/websock_adapter">Websock Adapter</a> which is kind-of like Plug for websockets.</p>
<p>Not only did Mat build Bandit and <a href="https://github.com/mtrudel/thousand_island">Thousand Island</a>, he built the Websock and Websock Adapter, then worked on integrating it with Phoenix. What a hero!</p>
<p><code class="language-plaintext highlighter-rouge">Websock</code> makes custom websocket integration way more flexible in Phoenix. Previously the only sane<sup id="fnref:8" role="doc-noteref"><a href="#fn:8" class="footnote" rel="footnote">8</a></sup> way of doing so was to implement a <code class="language-plaintext highlighter-rouge">Phoenix.Socket.Transport</code> and add it to the endpoint, as I described <a href="/elixir/phoenix/tutorial/2021/02/19/binary-websockets-with-elixir-phoenix.html">here</a>. Now we have the option to call <a href="https://hexdocs.pm/websock_adapter/WebSockAdapter.html#upgrade/4"><code class="language-plaintext highlighter-rouge">WebSockAdapter.upgrade/4</code></a> within a Plug or even a Phoenix Controller. This gives full access to the <a href="https://hexdocs.pm/plug/Plug.Conn.html">connection</a> before upgrade; only limited information is passed to the <a href="https://hexdocs.pm/phoenix/1.7.2/Phoenix.Socket.Transport.html#c:connect/1"><code class="language-plaintext highlighter-rouge">Phoenix.Transport.Connect.connect/1</code></a> implementation.</p>
<p>I wish I had known about <code class="language-plaintext highlighter-rouge">Websock</code> with Phoenix 1.7 earlier. <a href="https://hexdocs.pm/fedecks_server/readme.html">Fedecks Server</a> currently uses the <code class="language-plaintext highlighter-rouge">Phoenix.Socket.Transport</code> approach and a customer header for authentication on connection. This opens the opportunity for a cleaner approach.</p>
<h2 id="dont-fight-the-monolith--peter-ullrich">Don’t Fight the Monolith — Peter Ullrich</h2>
<p>This was ostensibly about techniques for keeping monolithic architectures healthy but was more about defining your <a href="https://www.martinfowler.com/bliki/BoundedContext.html">Bounded Contexts</a> using <a href="https://en.wikipedia.org/wiki/Domain-driven_design">DDD</a>’s <a href="https://en.wikipedia.org/wiki/Event_storming">Event Storming</a> and their relationships with <a href="https://www.infoq.com/articles/ddd-contextmapping/">Context Mapping</a>. This was fine but a bit of a bait and switch.</p>
<p>Peter did touch on two experiences: one with a company that found extracting their monolith into microservices<sup id="fnref:9" role="doc-noteref"><a href="#fn:9" class="footnote" rel="footnote">9</a></sup> and his current company that does not. He thought that a monolith is better for a company that is in “hypergrowth” and services for those that are more mature. I’m a little unconvinced of the argument.</p>
<p>Peter did also mention that their code is organised by bounded context / team, by namespace. He implied that the method of communication between contexts is controlled. I imagine that one team can not reach into the innards of the another context’s code but must (by enforced convention) go through specific “API modules”<sup id="fnref:10" role="doc-noteref"><a href="#fn:10" class="footnote" rel="footnote">10</a></sup> or possibly further disintermediation; I would also guess (hope) that direction of communication is also enforced. I would roll my eyes if they turn out to be loading the API implementation module for each boundary from config.</p>
<p>They have considered Saša Jurić’s <a href="https://hexdocs.pm/boundary/Boundary.html">Boundary package</a> but have not used it yet; I suspect that this may be a universal experience.</p>
<h2 id="change-data-capture-with-elixir-and-debezium--michal-gibowski-and-vanessa-loviton">Change Data Capture with Elixir and Debezium — Michal Gibowski and Vanessa Loviton</h2>
<p>I was reminded of the existence of database transaction logs (tlogs), and how tlogs propagation from the primary database is used to keep replica databases in sync. I learned that <a href="https://github.com/debezium/debezium">Debezium</a> is a project that can listen in on transaction logs, converting the tlog to a common format.</p>
<p>They use this approach as a <a href="https://martinfowler.com/bliki/StranglerFigApplication.html">Strangler Fig</a> for replacing a legacy application, including changing the structure of the database. Debezium is able to send changes to the database down to the new database over Kafka so it can be kept up to date with changes made through the old application. Pretty clever!</p>
<p>One issue mentioned is that each table is sent over a different topic. Sometimes child records can be received before the parent is created. They curently deal with this with a sleep and retry as it is not a bottleneck, but they could do something more sophisticated.</p>
<h2 id="telemetry-now-what--zac-barnes">Telemetry: Now what? — Zac Barnes</h2>
<p>This was a good basic introduction to using Telemetry in your (specifically Phoenix) application.</p>
<h2 id="chris-mccord-keynote">Chris McCord Keynote</h2>
<p>As of time of writing, I think Chris’ talk is the only one video publicly indexed and is <a href="https://www.youtube.com/watch?v=FADQAnq0RpA">here</a>.</p>
<p>Chris started with a call out to <a href="https://fly.io">Fly.io</a> saying it provided a Heroku-like experience from before when Heroku stopped working on things. That was funny.</p>
<p>Chris focussed the rest of his talk on LiveView. When LiveView was first conceived ambitions were much less than what has now been achieved. Chris expected it be limited to basic Apps, but now it can support most things that we would previously have used a client-side Javascript framework for - but with a lot less effort.</p>
<p>He spent some time explaining <a href="https://hexdocs.pm/phoenix_live_view/Phoenix.Component.html#module-slots">Slots</a> in the new component model, how they came from <a href="https://hexdocs.pm/surface/Surface.html">Surface</a>, and how much more flexible it makes components.</p>
<p><a href="https://hexdocs.pm/phoenix_live_view/live-navigation.html">Live Navigation</a> is a feature that they do not talk about enough - loading page changes over a websocket makes things more efficient.</p>
<p><a href="https://hexdocs.pm/phoenix_live_view/uploads.html">Uploads</a> are also easy and great, and allow uploading.</p>
<p>Other things included an infinite scrolling demo, “streams” for avoiding keeping large lists in memory, solving the previous issues with dynamically sized embedded items in a form. The latter involves very cheeky use of checkboxes.</p>
<p>Chris’s talk was the final one. I really enjoyed the conference. There were great talks and I enjoyed connecting an reconnecting with people.</p>
<p>Elixir Forum thread for any comments is <a href="https://elixirforum.com/t/blog-post-elixirconf-eu-2023/55610">here</a></p>
<hr />
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>I may write about this one day. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>Which had passed me by up until now(!). <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p><a href="https://blackadderquotes.com/i-have-a-cunning-plan">Blackadder</a> <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>I think <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p>Hexicles <a href="#fnref:5" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p>You could argue that is is true of any software or framework. It may be that Phoenix tends to a better architecture than others; it would be less true of buying / renting a service. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:7" role="doc-endnote">
<p>As far as I remember <a href="#fnref:7" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:8" role="doc-endnote">
<p>Alternatively you could implement Cowboy’s websocket handler yourself, which I have done. I found it too hard to configure this within Phoenix though so would end up creating a separate endpoint with <a href="https://hexdocs.pm/plug_cowboy/Plug.Cowboy.html">Plug Cowboy</a> and have to manage listening on different ports. Bit of a nightmare. <a href="#fnref:8" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:9" role="doc-endnote">
<p>I think for the purposes of this talk <em>microservices</em> mean <em>services</em>, ie not so micro <a href="#fnref:9" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:10" role="doc-endnote">
<p>Application in DDD terminology - but that gets confusing in BEAM <a href="#fnref:10" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Paul WilsonConference experience reports used to be a thing, back in the days of people doing lots of blogging. Why not be a bit retro, and put one together?Elixir mock (test double) testing seams: a modest proposal2023-03-24T10:56:13+00:002023-03-24T10:56:13+00:00/elixir/tdd/mocks/2023/03/24/elixir-mock-stub--fake-testing-seams-a-modest-proposal<p>The José Valim approved (tm) way of introducing mocks<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup> into Elixir is through <a href="https://dashbit.co/blog/mocks-and-explicit-contracts">injecting implementations of explicit contracts defined by behaviours</a>. José and pals crystallised this approach with the popular <a href="https://hexdocs.pm/mox/Mox.html">Mox hexicle</a>.</p>
<p>The standard way of injecting the mock or real implementation into the code under test is by passing modules around by some method. The implementation module is typically loaded from <em>application config</em> which can be tailored to the <em>mix environment</em>. I find this approach somewhat dissatisfying as the <em>module</em> being passed around is just an atom containing no metadata.</p>
<p>Apart from a general lack of tidiness, this provides a route for errors to slip through code which passes all the test. I do have a suggestion which would help. It will involve code that looks like this</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nv">@implementation</span> <span class="k">if</span> <span class="no">Mix</span><span class="o">.</span><span class="n">env</span><span class="p">()</span> <span class="o">==</span> <span class="ss">:test</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="no">MockCatFactsApi</span><span class="p">,</span> <span class="k">else</span><span class="p">:</span> <span class="no">RealCatFactsApi</span>
<span class="k">defmacro</span> <span class="n">__using__</span><span class="p">(</span><span class="n">_</span><span class="p">)</span> <span class="k">do</span>
<span class="kn">quote</span> <span class="k">do</span>
<span class="n">alias</span> <span class="kn">unquote</span><span class="p">(</span><span class="nv">@implementation</span><span class="p">),</span> <span class="ss">as:</span> <span class="no">CatFactsApi</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>You may not be attracted by that, but please bear with me. You may, at least, learn something surprising about feline collar bones. (I don’t believe the thing about Newton, though.)</p>
<h3 id="the-usual-approach">The usual approach</h3>
<p>Let’s get some cat facts using the standard method of implementation injection. The full implementation is <a href="https://github.com/paulanthonywilson/cat_facts">here</a>.</p>
<p>We’ll define a behaviour as a testing seam</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">CatFacts</span><span class="o">.</span><span class="no">CatFactsApi</span> <span class="k">do</span>
<span class="nv">@callback</span> <span class="n">get_facts</span><span class="p">(</span><span class="n">path</span> <span class="p">::</span> <span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">(),</span> <span class="n">finch_pool</span> <span class="p">::</span> <span class="n">atom</span><span class="p">)</span> <span class="p">::</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="no">Finch</span><span class="o">.</span><span class="no">Response</span><span class="o">.</span><span class="n">t</span><span class="p">()}</span> <span class="o">|</span> <span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="no">Exception</span><span class="o">.</span><span class="n">t</span><span class="p">()}</span>
<span class="k">end</span>
</code></pre></div></div>
<p>And we’re going to drive out our behaviour with tests.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">CatFactsTest</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span>
<span class="kn">import</span> <span class="no">Mox</span>
<span class="n">setup</span> <span class="ss">:verify_on_exit!</span>
<span class="n">test</span> <span class="s2">"Can get a fact"</span> <span class="k">do</span>
<span class="n">expect</span><span class="p">(</span><span class="no">MockCatFactsApi</span><span class="p">,</span> <span class="ss">:get_facts</span><span class="p">,</span> <span class="k">fn</span> <span class="s2">"fact"</span><span class="p">,</span> <span class="no">CatFinch</span> <span class="o">-></span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span>
<span class="p">%</span><span class="no">Finch</span><span class="o">.</span><span class="no">Response</span><span class="p">{</span>
<span class="ss">body:</span> <span class="s2">"{</span><span class="se">\"</span><span class="s2">fact</span><span class="se">\"</span><span class="s2">:</span><span class="se">\"</span><span class="s2">Cats are really dogs in disguise.</span><span class="se">\"</span><span class="s2">,</span><span class="se">\"</span><span class="s2">length</span><span class="se">\"</span><span class="s2">:33}"</span><span class="p">,</span>
<span class="ss">status:</span> <span class="mi">200</span>
<span class="p">}}</span>
<span class="k">end</span><span class="p">)</span>
<span class="n">assert</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="s2">"Cats are really dogs in disguise."</span><span class="p">}</span> <span class="o">==</span> <span class="no">CatFacts</span><span class="o">.</span><span class="n">fact</span><span class="p">()</span>
<span class="k">end</span>
<span class="c1"># After this we will want to test out verious error and edge conditions but </span>
<span class="c1"># we'll leave those out of here for brevity</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Inject mock and real implementations, with the private function <code class="language-plaintext highlighter-rouge">get_cat_facts_api/0</code>.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">CatFacts</span> <span class="k">do</span>
<span class="k">def</span> <span class="n">fact</span> <span class="k">do</span>
<span class="s2">"fact"</span>
<span class="o">|></span> <span class="n">cat_facts_api</span><span class="p">()</span><span class="o">.</span><span class="n">get_facts</span><span class="p">(</span><span class="no">CatFinch</span><span class="p">)</span>
<span class="o">|></span> <span class="n">handle_response</span><span class="p">()</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">cat_facts_api</span> <span class="k">do</span>
<span class="no">Application</span><span class="o">.</span><span class="n">get_env</span><span class="p">(</span><span class="ss">:cat_facts</span><span class="p">,</span> <span class="no">CatFacts</span><span class="o">.</span><span class="no">CatFactsApi</span><span class="p">,</span> <span class="no">CatFacts</span><span class="o">.</span><span class="no">RealCatFactsApi</span><span class="p">)</span>
<span class="k">end</span>
<span class="c1"># handle_response/2 ommited for brevity</span>
<span class="k">end</span>
</code></pre></div></div>
<p>If we define the <code class="language-plaintext highlighter-rouge">mock</code> (say in “test/support/mocks.ex”) and configure it for test then our tests will run.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Mox</span><span class="o">.</span><span class="n">defmock</span><span class="p">(</span><span class="no">MockCatFactsApi</span><span class="p">,</span> <span class="ss">for:</span> <span class="no">CatFacts</span><span class="o">.</span><span class="no">CatFactsApi</span><span class="p">)</span>
</code></pre></div></div>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># config/config.exs</span>
<span class="kn">import</span> <span class="no">Config</span>
<span class="n">import_config</span> <span class="s2">"</span><span class="si">#{</span><span class="n">config_env</span><span class="p">()</span><span class="si">}</span><span class="s2">.exs"</span>
</code></pre></div></div>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># config/test.exs</span>
<span class="kn">import</span> <span class="no">Config</span>
<span class="n">config</span> <span class="ss">:cat_facts</span><span class="p">,</span> <span class="no">CatFacts</span><span class="o">.</span><span class="no">CatFactsApi</span><span class="p">,</span> <span class="no">MockCatFactsApi</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cat_facts (main) $ mix test
....
Finished in 0.02 seconds (0.00s async, 0.02s sync)
4 tests, 0 failures
Randomized with seed 566481
</code></pre></div></div>
<p>Nearly there. We also need an actual implementation.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">CatFacts</span><span class="o">.</span><span class="no">RealCatFactsApi</span> <span class="k">do</span>
<span class="nv">@cat_facts_base</span> <span class="s2">"https://catfact.ninja"</span>
<span class="k">def</span> <span class="n">get_facts</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">finch_pool</span><span class="p">)</span> <span class="k">do</span>
<span class="n">url</span> <span class="o">=</span> <span class="no">Path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nv">@cat_facts_base</span><span class="p">,</span> <span class="n">path</span><span class="p">)</span>
<span class="ss">:get</span>
<span class="o">|></span> <span class="no">Finch</span><span class="o">.</span><span class="n">build</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
<span class="o">|></span> <span class="no">Finch</span><span class="o">.</span><span class="n">request</span><span class="p">(</span><span class="n">finch_pool</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Let’s get a cat fact!</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Erlang/OTP 25 [erts-13.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit]
Interactive Elixir (1.14.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> CatFacts.fact()
{:ok,
"The cat's clavicle, or collarbone, does not connect with other bones but is buried in the muscles of the shoulder region. This lack of a functioning collarbone allows them to fit through any opening the size of their head."}
</code></pre></div></div>
<p> </p>
<p>I did not know that about cat’s clavicles.</p>
<p>So, this is all great, but your company’s Star Chamber of Staff Engineers have just decreed a new coding standard: api functions must be alliterative. As the annual performance review is looming, we rush to rename.</p>
<h3 id="the-code-change">The code change</h3>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">CatFacts</span><span class="o">.</span><span class="no">CatFactsApi</span> <span class="k">do</span>
<span class="nv">@callback</span> <span class="n">fetch_fun_feline_facts</span><span class="p">(</span><span class="n">path</span> <span class="p">::</span> <span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">(),</span> <span class="n">finch_pool</span> <span class="p">::</span> <span class="n">atom</span><span class="p">)</span> <span class="p">::</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="no">Finch</span><span class="o">.</span><span class="no">Response</span><span class="o">.</span><span class="n">t</span><span class="p">()}</span> <span class="o">|</span> <span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="no">Exception</span><span class="o">.</span><span class="n">t</span><span class="p">()}</span>
<span class="k">end</span>
</code></pre></div></div>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">CatFactsTest</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span>
<span class="kn">import</span> <span class="no">Mox</span>
<span class="n">setup</span> <span class="ss">:verify_on_exit!</span>
<span class="n">test</span> <span class="s2">"Can get a fact"</span> <span class="k">do</span>
<span class="n">expect</span><span class="p">(</span><span class="no">MockCatFactsApi</span><span class="p">,</span> <span class="ss">:fetch_fun_feline_facts</span><span class="p">,</span> <span class="k">fn</span> <span class="s2">"fact"</span><span class="p">,</span> <span class="no">CatFinch</span> <span class="o">-></span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span>
<span class="p">%</span><span class="no">Finch</span><span class="o">.</span><span class="no">Response</span><span class="p">{</span>
<span class="ss">body:</span> <span class="s2">"{</span><span class="se">\"</span><span class="s2">fact</span><span class="se">\"</span><span class="s2">:</span><span class="se">\"</span><span class="s2">Cats are really dogs in disguise.</span><span class="se">\"</span><span class="s2">,</span><span class="se">\"</span><span class="s2">length</span><span class="se">\"</span><span class="s2">:33}"</span><span class="p">,</span>
<span class="ss">status:</span> <span class="mi">200</span>
<span class="p">}}</span>
<span class="k">end</span><span class="p">)</span>
<span class="n">assert</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="s2">"Cats are really dogs in disguise."</span><span class="p">}</span> <span class="o">==</span> <span class="no">CatFacts</span><span class="o">.</span><span class="n">fact</span><span class="p">()</span>
<span class="k">end</span>
<span class="c1"># still taking the other tests as read</span>
<span class="k">end</span>
</code></pre></div></div>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">CatFacts</span> <span class="k">do</span>
<span class="c1"># ...</span>
<span class="k">def</span> <span class="n">fact</span> <span class="k">do</span>
<span class="s2">"fact"</span>
<span class="o">|></span> <span class="n">cat_facts_api</span><span class="p">()</span><span class="o">.</span><span class="n">fetch_fun_feline_facts</span><span class="p">(</span><span class="no">CatFinch</span><span class="p">)</span>
<span class="o">|></span> <span class="n">handle_response</span><span class="p">()</span>
<span class="k">end</span>
<span class="c1"># etc...</span>
<span class="k">end</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cat_facts (main) $ mix test
Compiling 2 files (.ex)
....
Finished in 0.01 seconds (0.00s async, 0.01s sync)
4 tests, 0 failures
</code></pre></div></div>
<p>Phew! We’re done. Except, oh no! There’s an error in production.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cat_facts (main) $ iex -S mix
Erlang/OTP 25 [erts-13.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit]
Compiling 4 files (.ex)
Generated cat_facts app
Interactive Elixir (1.14.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> CatFacts.fact()
** (UndefinedFunctionError) function CatFacts.RealCatFactsApi.fetch_fun_feline_facts/2 is undefined or private
(cat_facts 0.1.0) CatFacts.RealCatFactsApi.fetch_fun_feline_facts("fact", CatFinch)
(cat_facts 0.1.0) lib/cat_facts.ex:9: CatFacts.fact/0
iex:1: (file)
</code></pre></div></div>
<p>Obviously you, perceptive reader, have already spotted both errors:</p>
<ol>
<li>We (ok I) forgot to add <code class="language-plaintext highlighter-rouge">@behaviour CatFacts.CatFactsApi</code></li>
<li>And we did not rename <code class="language-plaintext highlighter-rouge">CatFacts.get_facts/1</code></li>
</ol>
<p>You spotted it but the compiler did not; there were no warnings. Dialyzer can not help you either. The most you could say about <code class="language-plaintext highlighter-rouge">CatFacts.cat_facts_api/1</code> is that it returns an atom; that is returns an atom representing a module that implements a specific behaviour is an unsayable concept.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="c1"># not very helpful</span>
<span class="nv">@spec</span> <span class="n">cat_facts_api</span> <span class="p">::</span> <span class="n">atom</span><span class="p">()</span>
</code></pre></div></div>
<p>You may be thinking that this is a contrived example: this is not the kind of error that would be written and get past a code review.</p>
<p><a href="https://xkcd.com/908/"><img src="/assets/the_cropped_cloud.png" alt="Last two frames cropped from XKCD 908, The Cloud. Blackhat is sitting at a computer and cueball is asking questions.
Cueball: Should the cord be stretched across the room like this?
Blackhat: Of course. It has to reach the server and the server is is over there.
Cueball: What if someone trips on it?
Blackhat: Who would want to do that? It sounds unpleasant.
Cueball: Uh. Sometimes people do stuff by accident.
Blackhat: I don't think I know anybody like that.
" title="Last two frames cropped from XKCD 908, The Cloud. Blackhat is sitting at a computer and cueball is asking questions.
Cueball: Should the cord be stretched across the room like this?
Blackhat: Of course. It has to reach the server and the server is is over there.
Cueball: What if someone trips on it?
Blackhat: Who would want to do that? It sounds unpleasant.
Cueball: Uh. Sometimes people do stuff by accident.
Blackhat: I don't think I know anybody like that." /></a></p>
<p>Ok, Blackhat, my experience is a little different. I definitely found many examples of <code class="language-plaintext highlighter-rouge">@behaviour</code> being missed in a particular company’s codebase.</p>
<p>This kind of error can happen. It would be lovely if we could at least get some kind of compiler warning when renaming (or function arity changing) goes wrong. Read on to find out how we can.</p>
<h3 id="the-alternative-approach">The alternative approach</h3>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">CatFacts</span><span class="o">.</span><span class="no">CatFactsApi</span> <span class="k">do</span>
<span class="nv">@callback</span> <span class="n">fetch_fun_feline_facts</span><span class="p">(</span><span class="n">path</span> <span class="p">::</span> <span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">(),</span> <span class="n">finch_pool</span> <span class="p">::</span> <span class="n">atom</span><span class="p">)</span> <span class="p">::</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="no">Finch</span><span class="o">.</span><span class="no">Response</span><span class="o">.</span><span class="n">t</span><span class="p">()}</span> <span class="o">|</span> <span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="no">Exception</span><span class="o">.</span><span class="n">t</span><span class="p">()}</span>
<span class="k">defmacro</span> <span class="n">alias</span> <span class="k">do</span>
<span class="n">implementation</span> <span class="o">=</span> <span class="no">Application</span><span class="o">.</span><span class="n">get_env</span><span class="p">(</span><span class="ss">:cat_facts</span><span class="p">,</span> <span class="bp">__MODULE__</span><span class="p">,</span> <span class="no">CatFacts</span><span class="o">.</span><span class="no">RealCatFactsApi</span><span class="p">)</span>
<span class="kn">quote</span> <span class="k">do</span>
<span class="n">alias</span> <span class="kn">unquote</span><span class="p">(</span><span class="n">implementation</span><span class="p">),</span> <span class="ss">as:</span> <span class="no">CatFactsApi</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">CatFacts</span> <span class="k">do</span>
<span class="kn">require</span> <span class="no">CatFacts</span><span class="o">.</span><span class="no">CatFactsApi</span>
<span class="no">CatFacts</span><span class="o">.</span><span class="no">CatFactsApi</span><span class="o">.</span><span class="n">alias</span><span class="p">()</span>
<span class="k">def</span> <span class="n">fact</span> <span class="k">do</span>
<span class="s2">"fact"</span>
<span class="o">|></span> <span class="no">CatFactsApi</span><span class="o">.</span><span class="n">fetch_fun_feline_facts</span><span class="p">(</span><span class="no">CatFinch</span><span class="p">)</span>
<span class="o">|></span> <span class="n">handle_response</span><span class="p">()</span>
<span class="k">end</span>
<span class="c1"># etc...</span>
<span class="k">end</span>
</code></pre></div></div>
<p>The tests still run.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cat_facts (main) $ mix test
Compiling 3 files (.ex)
....
Finished in 0.01 seconds (0.00s async, 0.01s sync)
4 tests, 0 failures
</code></pre></div></div>
<p>But if we compile for <code class="language-plaintext highlighter-rouge">dev</code> or <code class="language-plaintext highlighter-rouge">prod</code> we get a warning. This should be a welcome safety net, especially if your build server is configured to treat warnings as errors.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cat_facts (main) $ mix compile
Compiling 2 files (.ex)
warning: CatFacts.RealCatFactsApi.fetch_fun_feline_facts/2 is undefined or private
lib/cat_facts.ex:13: CatFacts.fact/0
</code></pre></div></div>
<p>Incidentally, dialyzer is now similarly unimpressed</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
lib/cat_facts.ex:12:call_to_missing
Call to missing or private function CatFacts.RealCatFactsApi.fetch_fun_feline_facts/2.
________________________________________________________________________________
done (warnings were emitted)
Halting VM with exit status 2
</code></pre></div></div>
<p>There’s still some room for improvement.</p>
<ul>
<li>Having to <code class="language-plaintext highlighter-rouge">require CatFacts.CatFactsApi</code> before calling the <code class="language-plaintext highlighter-rouge">alias/0</code> macro is a bit awkward. My preference is to sidestep this a bit with <code class="language-plaintext highlighter-rouge">use</code>.</li>
<li>We’re using <code class="language-plaintext highlighter-rouge">Application.get_env/3</code> at compile time but we can’t use <code class="language-plaintext highlighter-rouge">Application.compile_env/3</code> inside a macro. We could use this with an attribute <code class="language-plaintext highlighter-rouge">@impl Application.compile_env(:cat_facts, __MODULE__, CatFacts.RealCatFactsApi)</code> but …</li>
<li>We are always using one implementation in the <code class="language-plaintext highlighter-rouge">test</code> environment and another elsewhere. I do not consider that configuration. My preference is to explicitly state the implementations in the code rather than having to look in another file (“config/test.exs”).</li>
</ul>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">CatFacts</span><span class="o">.</span><span class="no">CatFactsApi</span> <span class="k">do</span>
<span class="nv">@callback</span> <span class="n">fetch_fun_feline_facts</span><span class="p">(</span><span class="n">path</span> <span class="p">::</span> <span class="no">String</span><span class="o">.</span><span class="n">t</span><span class="p">(),</span> <span class="n">finch_pool</span> <span class="p">::</span> <span class="n">atom</span><span class="p">)</span> <span class="p">::</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="no">Finch</span><span class="o">.</span><span class="no">Response</span><span class="o">.</span><span class="n">t</span><span class="p">()}</span> <span class="o">|</span> <span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="no">Exception</span><span class="o">.</span><span class="n">t</span><span class="p">()}</span>
<span class="nv">@implementation</span> <span class="k">if</span> <span class="no">Mix</span><span class="o">.</span><span class="n">env</span><span class="p">()</span> <span class="o">==</span> <span class="ss">:test</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="no">MockCatFactsApi</span><span class="p">,</span> <span class="k">else</span><span class="p">:</span> <span class="no">CatFacts</span><span class="o">.</span><span class="no">RealCatFactsApi</span>
<span class="k">defmacro</span> <span class="n">__using__</span><span class="p">(</span><span class="n">_</span><span class="p">)</span> <span class="k">do</span>
<span class="kn">quote</span> <span class="k">do</span>
<span class="n">alias</span> <span class="kn">unquote</span><span class="p">(</span><span class="nv">@implementation</span><span class="p">),</span> <span class="ss">as:</span> <span class="no">CatFactsApi</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">CatFacts</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">CatFacts</span><span class="o">.</span><span class="no">CatFactsApi</span>
<span class="c1"># etc ..</span>
<span class="k">end</span>
</code></pre></div></div>
<h2 id="drawbacks">Drawbacks</h2>
<p>There are two potential drawbacks to the approach I am suggesting here:-</p>
<h3 id="reduction-in-flexibility">Reduction in flexibility</h3>
<p>This alternative approach needs the implementation switching to be happen at compile time. The standard approach allows for the implementation to be determined at runtime. This matters little when injecting mocks. It is possible you may want to use the pattern for other purposes, such as switching out an actual implementation in production controlled by a feature flag; in this case I would agree that something along the lines of passing modules around is still a reasonable approach.</p>
<h3 id="reviewers">Reviewers</h3>
<p>Asynchronous code reviews via pull requests have become ubiquitous over the last ten or so years. Combined with <a href="https://en.wikipedia.org/wiki/Big_Tech">FAANG</a>-style performance reviews, one of the side effects is presure on developers to review quickly while being able to display their own knowledge and competence<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup>.</p>
<p>Finding a macro in the code can be like catnip<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote" rel="footnote">3</a></sup> to a reviewing developer under those pressures. Without much thought they can block and say something to the effect of “This is over-engineering. You can replace these 3 simple lines of code with these other lines of code.” If they are feeling particularly pompous they might just quote from the macro or <a href="https://hexdocs.pm/elixir/1.12/library-guidelines.html#avoid-macros">library guidelines</a>.</p>
<p>If I have managed to persuade you to give this method of implementation-injection a shot, your ability to to actually do so may be limited by the social and power dynamics in your organisation.</p>
<h2 id="advantages">Advantages</h2>
<h3 id="better-mistake-cover-from-compiler-or-dialyzer-warnings">Better mistake cover from compiler or dialyzer warnings</h3>
<p>I hope that I have already established this advantage. I should probably point out that the alternative approach will not let you know about forgotten <code class="language-plaintext highlighter-rouge">@behaviour</code> directives, but it will protect you from the consequences.</p>
<h3 id="less-bloated-configuration">Less bloated configuration</h3>
<p>Large projects that make heavy use of <code class="language-plaintext highlighter-rouge">Mox</code> often end up massive “text.exs” files which are a headache to organise and maintain.</p>
<p>If something can be known at compile time and never changes between different physical environment (ie different developer’s laptops, build servers, deployments) then I do not think that is configuration. If you can inline that to where it is used then you have made things simpler and more explicit.</p>
<p>It might look odd to you (and unfortunately your reviewers) at first. That is because you (and they) are not used to it.</p>
<h2 id="more-cat-facts">More cat facts</h2>
<p>While were here, and having fixed the implementation …</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="p">(</span><span class="mi">7</span><span class="p">)</span><span class="o">></span> <span class="no">CatFacts</span><span class="o">.</span><span class="n">fact</span><span class="p">()</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span>
<span class="s2">"When a cat drinks, its tongue - which has tiny barbs on it - scoops the liquid up backwards."</span><span class="p">}</span>
<span class="n">iex</span><span class="p">(</span><span class="mi">8</span><span class="p">)</span><span class="o">></span> <span class="no">CatFacts</span><span class="o">.</span><span class="n">fact</span><span class="p">()</span>
</code></pre></div></div>
<p>Wait? What does scooping up liquid backwards even mean? Let’s try another one.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">iex</span><span class="p">(</span><span class="mi">16</span><span class="p">)</span><span class="o">></span> <span class="no">CatFacts</span><span class="o">.</span><span class="n">fact</span><span class="p">()</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span>
<span class="s2">"Isaac Newton invented the cat flap. Newton was experimenting in a pitch-black room. Spithead, one of his cats, kept opening the door and wrecking his experiment. The cat flap kept both Newton and Spithead happy."</span><span class="p">}</span>
</code></pre></div></div>
<p>Citation needed and I doubt it, though I do feel better having read that.</p>
<p>Even if true, it still would not not be my favourite cat fact. My favourite is one that I read on an information board at <a href="https://www.highlandwildlifepark.org.uk">The Highland Wildlife Park</a>: the decline in Scottish wildcat numbers was reduced during the First World War because conscription reduced the gamekeeper population.</p>
<p><img src="/assets/wild_cat_ww1.png" alt="Photo of information board: "The prolonged tragedy of WW1 calls up gamekeepers leading to a decline in wildcat persecution" /></p>
<p>PS Just saw another cat fact in the news as I was writing this: approval for releasing <a href="https://www.bbc.co.uk/news/uk-scotland-highlands-islands-65065167">Scottish wildcats being into The Cairngorms has been granted</a> from The Highland Wildlife Park.</p>
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>There’s some awkward terminology around all this which is probably not important. I should really say <em>test double</em> but I’ve always found that an awkward phrase. See <a href="http://xunitpatterns.com/Mocks,%20Fakes,%20Stubs%20and%20Dummies.html">XUnit patters</a> for definitions. It is common in Elixir Land (as other places) to use <em>Mock</em> for <em>Test Doubles</em> so I will just stick with that here; being more correct would also be more confusing. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>It’s a hard trap to avoid. I’ve been meaning to write something about reviewing more effectively but I doubt I could better <a href="https://cultivatehq.com/posts/how-to-be-a-kinder-more-effective-code-reviewer/">Dan Munckton</a> or <a href="https://chelseatroy.com/2019/12/18/reviewing-pull-requests/">Chelsea Troy</a>. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>Cat fact. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Paul WilsonThe José Valim approved (tm) way of introducing mocks1 into Elixir is through injecting implementations of explicit contracts defined by behaviours. José and pals crystallised this approach with the popular Mox hexicle. There’s some awkward terminology around all this which is probably not important. I should really say test double but I’ve always found that an awkward phrase. See XUnit patters for definitions. It is common in Elixir Land (as other places) to use Mock for Test Doubles so I will just stick with that here; being more correct would also be more confusing. ↩A fun, but trivial, limitation in Elixir ExUnit (/Macros)2023-03-10T11:59:43+00:002023-03-10T11:59:43+00:00/elixir/2023/03/10/a-fun-but-trivial-limitation-in-exunit-macros<p>Today I learnt that it is forbidden to use an attribute which is a <a href="https://www.erlang.org/doc/reference_manual/data_types.html#port-identifier">port</a> or a (<a href="https://www.erlang.org/doc/reference_manual/data_types.html#reference">reference</a>) in an <a href="https://hexdocs.pm/elixir/Port.html">ExUnit</a> test. (<a href="https://www.erlang.org/doc/reference_manual/data_types.html#pid">Pids</a> are just fine).</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">ScratchpadTest</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span>
<span class="nv">@port</span> <span class="ss">:erlang</span><span class="o">.</span><span class="n">list_to_port</span><span class="p">(</span><span class="s1">'#Port<0.1234>'</span><span class="p">)</span>
<span class="n">test</span> <span class="s2">"porty worty"</span> <span class="k">do</span>
<span class="nv">@port</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Gets you</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>** (ArgumentError) cannot inject attribute @port into function/macro because
cannot escape #Port<0.1234>. The supported values are: lists, tuples, maps,
atoms, numbers, bitstrings, PIDs and remote functions in the format &Mod.fun/arity
</code></pre></div></div>
<p>It is, of course, super-easy to work around:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">ScratchpadTest</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span>
<span class="n">test</span> <span class="s2">"porty worty"</span> <span class="k">do</span>
<span class="n">assert</span> <span class="n">is_port</span><span class="p">(</span><span class="ss">:erlang</span><span class="o">.</span><span class="n">list_to_port</span><span class="p">(</span><span class="s1">'#Port<0.1234>'</span><span class="p">))</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>You might, as I did, think the limitation was caused solely by <code class="language-plaintext highlighter-rouge">test/2</code> being a macro and I suppose it kind-of is. But it is easy to forget that Macros are everywhere in Elixir. This does not work.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">ScratchpadTest</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">ExUnit</span><span class="o">.</span><span class="no">Case</span>
<span class="nv">@port</span> <span class="ss">:erlang</span><span class="o">.</span><span class="n">list_to_port</span><span class="p">(</span><span class="s1">'#Port<0.1234>'</span><span class="p">)</span>
<span class="n">test</span> <span class="s2">"porty worty"</span> <span class="k">do</span>
<span class="n">assert</span> <span class="n">is_port</span><span class="p">(</span><span class="n">port</span><span class="p">())</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">port</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="nv">@port</span>
<span class="k">end</span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">defp/2</code> and <code class="language-plaintext highlighter-rouge">def/2</code> are also macros. This too does not compile</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">Scratchpad</span> <span class="k">do</span>
<span class="nv">@port</span> <span class="ss">:erlang</span><span class="o">.</span><span class="n">list_to_port</span><span class="p">(</span><span class="s1">'#Port<0.1234>'</span><span class="p">)</span>
<span class="k">def</span> <span class="n">port</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="nv">@port</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Weirdly though this does compile, even though <code class="language-plaintext highlighter-rouge">defmodule/2</code> is a macro.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">Scratchpad</span> <span class="k">do</span>
<span class="nv">@port</span> <span class="ss">:erlang</span><span class="o">.</span><span class="n">list_to_port</span><span class="p">(</span><span class="s1">'#Port<0.1234>'</span><span class="p">)</span>
<span class="no">IO</span><span class="o">.</span><span class="n">inspect</span><span class="p">(</span><span class="nv">@port</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div></div>
<p>No doubt if I was more learned, or put the research in, I would understand why. None of this is that important so</p>
<p>¯\(°_o)/¯☕️</p>
<p>Anyway, if you have read this far sorry (not sorry) for you learning nothing useful. I’m going back to some crazy test-driving of websocket client stuff using <a href="https://hexdocs.pm/mint_web_socket/Mint.WebSocket.html">Mint Websocket</a>.</p>
<hr />
<p><strong>Update 2023-03022</strong> Late update: ubiquitous and helpful Elixir Forum poster Benjamin Milde <a href="https://elixirforum.com/t/blog-post-a-fun-but-trivial-limitation-in-elixir-exunit-macros/54484/2?u=paulanthonywilson">explained the problem is the way Elixir macros work</a>: they essentially work by injecting the macro AST at the point in the code AST. As ports and reference values can not be seralised to AST then things fail. That makes sense in terms of mechanism and that there is no good reason for passing ports or references from compile time to runtime. (Although I now do wonder why PIDs are AST serialisable).</p>
<p>The actual issue is between compile and runtime, not really macros, which is obvious now I think about it. For example, this does not comile</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">Scratchpad</span> <span class="k">do</span>
<span class="nv">@port</span> <span class="ss">:erlang</span><span class="o">.</span><span class="n">list_to_port</span><span class="p">(</span><span class="s1">'#Port<0.1234>'</span><span class="p">)</span>
<span class="k">def</span> <span class="n">port</span><span class="p">,</span> <span class="k">do</span><span class="p">:</span> <span class="nv">@port</span>
<span class="k">end</span>
</code></pre></div></div>
<p>I should really rewrite this post with that emphasis.</p>Paul WilsonToday I learnt that it is forbidden to use an attribute which is a port or a (reference) in an ExUnit test. (Pids are just fine).I have released some Elixir Nerves Libraries2023-01-23T14:28:40+00:002023-01-23T14:28:40+00:00/nerves/elixir/2023/01/23/nerves-hexicletastic<p>I recently got back to a slow burn project to make a security system for my home-office, with <a href="https://nerves-project.org">Elixir Nerves</a>. Being one of my projects, it was an Umbrella Project. The last time I posted it was to <a href="/elixir/2022/09/14/leaving-the-umbrella-behind.html">(relucanctly) leave Umbrellas behind</a>, so I did think about flattening the project. Instead I realised that most of apps were generally reusable, at least by me<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>, so instead I have extracted them into their own <a href="https://mastodon.social/@paulwilson/109721602345750131">hexicles</a>.</p>
<h2 id="configuration-embracing-the-anti-pattern">Configuration: embracing the anti-pattern</h2>
<p>I tend to not configure too much within project code as it’s generally as easy to to change a module attribute as it is to change a value in a config file. I did feel like I had to make things a bit more configurable; someone else probably doesn’t want to be stuck with my random GPIO pin choices, for instance.</p>
<p>I went with the laziest choice of using application configuration in all the extractions so far, as in the library customisation method that is <a href="https://hexdocs.pm/elixir/1.14.3/library-guidelines.html#avoid-application-configuration">specifically called out as an antipattern in the Elixir Library Guidelines</a>. In general, I agree with that guideline<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup>. I’ve ignored it here as none of the extracted libraries could be used with multiple configurations, and it is the simplest to write and for clients to use.</p>
<h2 id="logging">Logging</h2>
<p>In general I am not too keen on libraries that log. I figure that it is better to find some way to communicate the information back to the client and let it decide what to do. Nerves stuff logs a lot though, so I figure why buck that trend and make things more complicated? So I have kept logging in.</p>
<h2 id="are-umbrella-projects-still-bad">Are Umbrella Projects still bad?</h2>
<p>I don’t know. The <a href="/elixir/2022/09/14/leaving-the-umbrella-behind.html">previous objections</a> still stand, but there is something to be said for easily writing code for a subsytem <em>in-situ</em> then extracting it out into its own library later. I should force myself to write a bit more about this.</p>
<h2 id="the-hexicles">THE HEXICLES!!!</h2>
<p>Here they are:</p>
<h3 id="vintage-heart">Vintage Heart</h3>
<ul>
<li><a href="https://github.com/paulanthonywilson/vintage_heart/">Github</a></li>
<li><a href="https://hexdocs.pm/vintage_heart/readme.html">Hexdocs</a></li>
<li><strong>Mix Install</strong> <code class="language-plaintext highlighter-rouge">{:vintage_heart, "~> 0.1.1"}</code></li>
</ul>
<p>I often seem to be plagued by intermittent loss of connectivity over WiFi. Everything will work for days, even weeks, and then the device will keep dropping offline. Intermittent things are really difficult to debug<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote" rel="footnote">3</a></sup>, compounded by it being difficult connect to an offline device.</p>
<p>Vintage Heart keeps an eye things by using <a href="https://hexdocs.pm/vintage_net/readme.html#internet-connectivity-checks">VintageNet connectivity checking</a> and takes action if offline for a period. By default it will</p>
<ul>
<li>Kick VintageNet after 4 minutes of being offline. By kick, I mean kill the <code class="language-plaintext highlighter-rouge">VintageNet.RouteManager</code> process, which normally does the trick<sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote" rel="footnote">4</a></sup>.</li>
<li>Causes a reboot after 14 minutes, during which time there would have been 3 failed kicks.</li>
</ul>
<h3 id="connectivity-led-status">Connectivity LED status</h3>
<ul>
<li><a href="https://github.com/paulanthonywilson/connectivity_led_status/">Github</a></li>
<li><a href="https://hexdocs.pm/connectivity_led_status/readme.html">Hexdocs</a></li>
<li><strong>Mix install</strong>: <code class="language-plaintext highlighter-rouge">{:connectivity_led_status, "~> 0.1.2"}</code></li>
</ul>
<p>On the connectivity theme, uses an onboard LED to indicate connectivity status.</p>
<ul>
<li>Flashes rapidly if no WiFi address is allocated</li>
<li>Flashes a heartbeat (2 rapid flashes then a pause) if a IP address is established but VintageNet is not reporting Internet connectivity</li>
<li>Flashes in a measured way when the VingateNetWizard is up</li>
<li>Does not flash when a WiFi address (other than the VintageNetWizard) is allocated and Network connectivity is established</li>
</ul>
<p>I find it useful to have a visual clue about how connected a device is.</p>
<h3 id="vintage-net-wizard-launcher">Vintage Net Wizard Launcher</h3>
<ul>
<li><a href="https://github.com/paulanthonywilson/vintage_net_wizard_launcher/">Github</a></li>
<li><a href="https://hexdocs.pm/vintage_net_wizard_launcher/readme.html">Hexdocs</a></li>
<li><strong>Mix install</strong>: <code class="language-plaintext highlighter-rouge">{:vintage_net_wizard_launcher, "~> 0.1.0"}</code></li>
</ul>
<p>The Vintage Net Wizard is a super-useful little <a href="https://hexdocs.pm/vintage_net_wizard/readme.html">hexicle</a> that configures the WiFi to AP mode and sets up a little webserver. Someone can then connect to the hotspot and configure the actual WiFi details.</p>
<p>This launches the wizard, so you don’t have to. By default:</p>
<ul>
<li>It launches the wizard on startup, if the WiFi is not configured</li>
<li>It launches the wizard if GPIO pin 21 is detected as being high (ie under voltage) for 3 seconds. For instance if you have connected 21 to 3V via a button, and pressed it for 3 seconds.</li>
</ul>
<h3 id="ds18b20">DS18B20</h3>
<ul>
<li><a href="https://github.com/paulanthonywilson/ds18b20">Github</a></li>
<li><strong>Mix install</strong>: <code class="language-plaintext highlighter-rouge">{:ds18b20, git: "git@github.com:paulanthonywilson/ds18b20.git}</code></li>
</ul>
<p>DS18b20 is a digital thermometer that uses the weird <a href="https://en.wikipedia.org/wiki/1-Wire">1-Wire</a> system. This library supports reading the temperature as long as you’ve gone through the <a href="http://www.carstenblock.org/post/project-excelsius/">shenannigans to enable 1-Wire</a>, and this is the only 1-Wire device used. For convenience <a href="https://github.com/paulanthonywilson/ds18b20/blob/main/lib/ds18b20/temperature_server.ex#L36-L41">you can subscribe</a> to get a reading every minute.</p>
<p>I haven’t published it to Hex, as there is now <a href="https://hex.pm/packages/ds18b20_1w">a project there ahead of me</a>. I haven’t tried it but it does seem better in that it supports multiple sensors.</p>
<h3 id="simplest-pub-sub">Simplest Pub Sub</h3>
<ul>
<li><a href="https://github.com/paulanthonywilson/simplest_pub_sub/">Github</a></li>
<li><a href="https://hexdocs.pm/simplest_pub_sub/readme.html">HexDocs</a></li>
<li><strong>Mix install</strong>: <code class="language-plaintext highlighter-rouge">{:simplest_pub_sub, "~> 0.1.0"}</code></li>
</ul>
<p>Simple wrapper of <a href="https://hexdocs.pm/elixir/Registry.html">Registry</a> for using Pub Sub. It’s really only there to save me writing the same 10 lines of code over and over again. I’m not writing in <a href="https://go.dev">Go</a>.</p>
<p>It also means that my extracted hexicles can share a pub-sub registry.</p>
<h3 id="next-up-movement-detection-for-determining-whether-a-room-is-occupied">Next up, movement detection for determining whether a room is occupied</h3>
<ul>
<li><a href="https://github.com/paulanthonywilson/hc_sr_501_occupation">Github</a></li>
<li><a href="https://hexdocs.pm/hc_sr_501_occupation/readme.html">HexDocs</a></li>
<li><strong>Mix install</strong>: <code class="language-plaintext highlighter-rouge">{:hc_sr_501_occupation, "~> 0.1.0"}</code></li>
</ul>
<p>An <a href="https://duckduckgo.com/?q=HC-SR501">HC-SR 501 infra-red sensor</a> is a cheap device for detecting movement. <del>Next I’ll extract out a hexicle to broadcast to subscribers</del>. This hexicle allows processes in your Nerves application to subscribe and receive updates:-</p>
<ul>
<li>when movement is detected</li>
<li>when movement is no longer detected</li>
<li>when an area has become unoccupied (ie movement has not been detected for a configurable amount of time)</li>
<li>when an unoccupied room becomes occupied</li>
</ul>
<p>In this case I have swapped <a href="https://hexdocs.pm/elixir/1.14.3/library-guidelines.html#avoid-application-configuration">Library Guideline Antipattern</a>. I figure it should be possible to attach multiple sensors to your Nerves device, so I have not sinned by adding library configuration. Instead I have committed the act of some <a href="https://github.com/paulanthonywilson/hc_sr_501_occupation/blob/e4706073c2c3f8c4b09c68337582faae61ca7e1a/lib/hc_sr501_occupation/movement_sensor.ex#L7">light meta-programming</a> to make things easier the anyone using this hexicle.</p>
<hr />
<p><strong>Update 2023-05-08</strong> Updated the HC-SR 501 section as I did release that hexicle ages ago(!)</p>
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>Don’t tell anybody but I have been known to entirely copy an Umbrella app from one Nerves project and paste it into another. Hey! They’re my projects; stop judging. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>Actually I’m not super keen on many of the other bits in the library guidelines. To me, it is often too prescriptive and is sprinkled with general programming (as opposed to library) advice. I should write something about it sometime. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>It may be an issue with my home network. I even occasionally get loss of connectivity on my phone when wandering between rooms; I suspect it’s to do with switching between access points but I ain’t no network engineer. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>I worked this out while connected to a Pi Zero W over USB, while the Pi was having WiFi issues. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Paul WilsonI recently got back to a slow burn project to make a security system for my home-office, with Elixir Nerves. Being one of my projects, it was an Umbrella Project. The last time I posted it was to (relucanctly) leave Umbrellas behind, so I did think about flattening the project. Instead I realised that most of apps were generally reusable, at least by me1, so instead I have extracted them into their own hexicles. Don’t tell anybody but I have been known to entirely copy an Umbrella app from one Nerves project and paste it into another. Hey! They’re my projects; stop judging. ↩Leaving the umbrella behind2022-09-14T14:26:54+00:002022-09-14T14:26:54+00:00/elixir/2022/09/14/leaving-the-umbrella-behind<p>I got into programming Elixir somewhere around 2013. (I’m not great with dates but I had a check of some projects on Github). Since I found out about them, every project I have initiated has been an <a href="https://elixir-lang.org/getting-started/mix-otp/dependencies-and-umbrella-projects.html#umbrella-projects">Umbrella Project</a>; for the past few years, though, I do seem to be the only person who likes Umbrellas<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>.</p>
<p>I have noticed that all projects that I’ve been brought in to work on are Flat<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup>, that I never see example code for Umbrellas (eg in deployment tutorials), and that there are occasional negative (but non-specific) negative comments about them on Elixir Forum. The only two blog posts that I could find that decry Umbrellas are <a href="https://embedded-elixir.com/post/2017-05-19-poncho-projects/">Gregg Mefford’s Nerves one introducing Poncho Apps</a> and <a href="https://www.jackmarchant.com/the-problem-with-elixir-umbrella-apps">The problem with Elixir Umbrella Apps, by Jack Marchant</a>. I find neither of those at all convincing, for reasons I describe later. This had me wondering if it were me that was out of touch, or whether every other Elixir programmer was wrong.</p>
<p><img src="/assets/skinner_out_of_touch.jpg" alt="Principal Skinner Out of Touch Meme" /></p>
<p>So I <a href="https://elixirforum.com/t/what-s-wrong-with-umbrella-apps/49585">asked on Elixir Forum</a> and got some answers that I did not like.</p>
<p>Turns out that I have been looking at this the wrong way. Rather than there being strong reasons that Umbrellas are bad, people do not experience much benefit from them. At the same time Umbrella Projects do add a certain amount of overhead and tooling issues. People do not find that Umbrellas entail many costs and few benefits. Reluctantly, I can see their point.</p>
<h2 id="why-i-like-umbrella-projects-in-the-first-place">Why I like Umbrella Projects in the first place</h2>
<p>At Nordic Ruby 2011 <a href="https://en.wikipedia.org/wiki/Tom_Preston-Werner">Tom Preston-Werner</a> crammed a lot of good stuff into his 30 minute talk<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote" rel="footnote">3</a></sup>, including the idea of separating out units of functionality in a (Ruby / Ruby on Rails) application <em>as if</em> they were <a href="https://en.wikipedia.org/wiki/RubyGems">Ruby Gems</a> even if it would never make sense to extract them into a <em>gem</em>. I loved that concept; it introduced a very clean way of seperating different areas of the domain<sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote" rel="footnote">4</a></sup>. Unfortunately when I came to try out the idea, the pain of going against the grain in Rails Apps proved too much.</p>
<p>When I discovered Umbrella Projects in Elixir I was delighted. They <em>just worked</em> out of the box to give neat seperation of concerns and enforcing directionality in intra-app dependencies. Each app is structured as if it were a separate <a href="https://twitter.com/paulanthonywils/status/1050442755833556992">hexicle</a>, making them beautifully separated. I love the neatness of keeping all the related files, even the tests, closely together.</p>
<p>As mentioned, when I have had the choice I’ve been (mostly) happily using Umbrellas ever since, and bemoaning their absence when working on <em>Flat Projects</em>.</p>
<h2 id="wrong-reasons-that-umbrella-projects-are-bad">Wrong reasons that Umbrella Projects are bad</h2>
<p>People have made a number of arguments against Umbrellas, some of which do not stack up.</p>
<h3 id="configuration-and-ponchos">Configuration and Ponchos</h3>
<p>Ponchos were introduced, or at least popularised, in the previously mentioned <a href="https://embedded-elixir.com/post/2017-05-19-poncho-projects/">very short Nerves Post</a>. To save you a click the issue identified was that Umbrella Projects default to pointing their config to <code class="language-plaintext highlighter-rouge">config/config.exs</code>, and that contained</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">import_config</span> <span class="s2">"../apps/*/config/config.exs"</span>
</code></pre></div></div>
<p>Configuration was loaded from all the Umbrella Projects in undefined order.</p>
<p>While the objection is no longer valid as these days Umbrella Projects default to using a single configuration, at the root of the Umbrella, it was not valid in 2017 either. Changing the root <code class="language-plaintext highlighter-rouge">config.exs</code> to do something different, such as using configuration at the root or loading any application configuration in a defined order was always trivially easy. Just so something is generated a certain way does not mean you can’t change it.</p>
<p>Poncho apps are simply standalone Elixir apps, that are linked using the <code class="language-plaintext highlighter-rouge">path</code> attribute in the dependencies eg (<code class="language-plaintext highlighter-rouge">{:myappdep, path: "../myappdep"}</code>), with a single designated “app” used to build releases (/ Nerves firmware).</p>
<p>I have not heard this, but an argument could be made for a slight advantage of Ponchos over an Umbrella: dependencies are compiled in the <code class="language-plaintext highlighter-rouge">prod</code> Mix.env() regardless of the environment being used to compile the main app. This makes it even more like using a <a href="https://twitter.com/paulanthonywils/status/1050442755833556992">hexicle</a>, for example <code class="language-plaintext highlighter-rouge">test/supprt</code> that is only compiled in <code class="language-plaintext highlighter-rouge">test</code> could not be shared between applications.</p>
<p>If you are definitely planning to extract and publish applications as separate <a href="https://twitter.com/paulanthonywils/status/1050442755833556992">hexicles</a>, a Poncho-like structure <em>might</em> be the way to go.</p>
<h3 id="that-other-blog-post">That other Blog post</h3>
<p>The (previously mentioned) <a href="https://www.jackmarchant.com/the-problem-with-elixir-umbrella-apps">other anti-Umbrella post</a> that I could find has also has some pretty week arguments mostly consisting of untrue or unsupported statements:</p>
<blockquote>
<p>if everything is deployed together you can still technically access modules that are technically circular dependencies, which kind of breaks the separation concept.</p>
</blockquote>
<p>Well, no. If you explicitly create a cyclic dependency in the <code class="language-plaintext highlighter-rouge">mix.exs</code>’s then the compilation will fail.</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">**</span> <span class="o">(</span>Mix<span class="o">)</span> Could not <span class="nb">sort </span>dependencies. There are cycles <span class="k">in </span>the dependency graph
</code></pre></div></div>
<p>Technically you <em>could</em> call from one app to another without explicitly making the other a dependency but you would have to ignore a <strong>massive</strong> warning, which <em>would</em> make you a fool.</p>
<blockquote>
<p>it will most likely slow you down the more code you add as the boundaries become more brittle and blurred</p>
</blockquote>
<p>Unless you are the kind of fool who ignores massive warnings, then the Umbrella Project related boundaries will remain crisp.</p>
<blockquote>
<p>Umbrella child apps are intended to be created as a way to deploy each of them separately</p>
</blockquote>
<p>Intended? Intended by whom? You could<sup id="fnref:5" role="doc-noteref"><a href="#fn:5" class="footnote" rel="footnote">5</a></sup> create separate <a href="https://elixir-lang.org/getting-started/mix-otp/config-and-releases.html#releases">Elixir releases</a> to deploy different parts of an Umbrella separately, but I have seen no evidence that is their purpose. The blog post contains no links, or supporting arguments, that back up that statement.</p>
<p>If you create a new Phoenix app with <code class="language-plaintext highlighter-rouge">mix new myapp --umbrella</code> then you get two applications which can not be deployed separately in any way that makes sense, which makes me extra-sceptical about this apparent intention of Umbrella Projects.</p>
<blockquote>
<p>I can guarantee moving into an umbrella app configuration, retrofitting on an existing app is the easier option than consolidating child apps</p>
</blockquote>
<p>That’s one of those statements that is hard to argue with, because it is unsupported. In my experience it is much easier to consolidate separated code than to separate consolidated code, because the dependencies in the latter invariably need a lot of unpicking.</p>
<h3 id="umbrellas-do-not-make-sense-for-code-organisation">Umbrellas do not make sense for code organisation</h3>
<blockquote>
<p>OTP applications are a runtime and deployment concern, not a code separation tool, so it doesn’t even make sense to use an umbrella for code organization in the first place</p>
</blockquote>
<p>From <a href="https://elixirforum.com/t/what-s-wrong-with-umbrella-apps/49585/11?u=paulanthonywilson">this Elixir Forum response</a>.</p>
<p>That specific point<sup id="fnref:6" role="doc-noteref"><a href="#fn:6" class="footnote" rel="footnote">6</a></sup> had me thinking for a while, because I use Umbrellas entirely for code organisation and never<sup id="fnref:7" role="doc-noteref"><a href="#fn:7" class="footnote" rel="footnote">7</a></sup> worry about separate deployments or starting up independently from other <em>applications</em>. But I reflected that, any <em>Mix project</em> an <em>app</em> in an Umbrella Project does not even need to be an <em>application</em> with its own supervision tree. Again, I don’t think the “not for code organisation” makes sense in itself.</p>
<h2 id="valid-reasons-not-to-use-umbrellas">Valid reasons not to use Umbrellas</h2>
<p>These are all covered in <a href="https://elixirforum.com/t/what-s-wrong-with-umbrella-apps/49585">that Elixir Forum thread</a>, but boil down to the advantages being few and achievable by other means while the disadvantages are several and impair both <a href="http://www.exampler.com/blog/2007/05/16/six-years-later-what-the-agile-manifesto-left-out/http://www.exampler.com/blog/2007/05/16/six-years-later-what-the-agile-manifesto-left-out/">ease and joy</a> in the programming.</p>
<p><a href="http://www.exampler.com/ease-and-joy/"><img src="/assets/disciplineskilleasejoy.jpg" alt="Agit Prop Style poster of raised chlenched fists with one ringing a bell. The title is "Discipline Skill Ease Joy"" /></a></p>
<h3 id="poor-tooling-support">Poor tooling support</h3>
<p>This can be quite a headache. Poor support for <code class="language-plaintext highlighter-rouge">mix xref</code> was cited a few times, eg <a href="https://elixirforum.com/t/what-s-wrong-with-umbrella-apps/49585/9?u=paulanthonywilson">here</a></p>
<p>A telling quote is</p>
<blockquote>
<p>Also the fact that you have to append “in an umbrella app” whenever you ask someone for advice on fixing your weird bug should be a sign that something is off from the start.</p>
</blockquote>
<p>From <a href="https://elixirforum.com/t/what-s-wrong-with-umbrella-apps/49585/11?u=paulanthonywilson">here</a></p>
<p>Something else that has also irritated me:</p>
<blockquote>
<p>various paths that are printed by mix tasks (e.g. test IIRC) aren’t “clickable” in vscode (I couldn’t click to open the file in the editor), because the printed paths are missing the apps/myapp/ prefix. I found this extremely annoying and disruptive.</p>
</blockquote>
<p><a href="https://elixirforum.com/t/what-s-wrong-with-umbrella-apps/49585/18?u=paulanthonywilson">From here</a></p>
<p>Supporting Umbrella Projects must be a pain for tool maintainers that need to care about the structure of your files, both when writing the code and testing the different scenarios. It is no wonder that support for Umbrellas is missing, or buggy. (I think that both my PRs to Nerves have been for Umbrella support).</p>
<p>A related issue not brought up (except by me) is that following tutorials needs an extra layer of translation for Umbrella Projects, eg in the <a href="https://fly.io/docs/elixir/getting-started/">Fly IO Phoenix Deployment documentation</a>.</p>
<h3 id="more-directories-to-navigate-and-more-files">More directories to navigate, and more files</h3>
<blockquote>
<p>they create an extra layer of indirection in your file paths (apps/ directory)</p>
</blockquote>
<p><a href="https://elixirforum.com/t/what-s-wrong-with-umbrella-apps/49585/3?u=paulanthonywilson">here</a></p>
<p>More directories can be a pain to navigate, especially if you are using a navigation tree. Having to be in different directories to perform different mix tasks can also be a bit of a headache.</p>
<p>Another problem with the multiple “apps” is the repeated identical file names. I have edited the wrong <code class="language-plaintext highlighter-rouge">mix.exs</code> or <code class="language-plaintext highlighter-rouge">application.ex</code> file on several occasions.</p>
<h3 id="more-code-slower-build">More code, slower build</h3>
<blockquote>
<p>about 2k LOC less, due to removal of the repetitive boilerplate across subprojects</p>
</blockquote>
<p><a href="https://elixirforum.com/t/what-s-wrong-with-umbrella-apps/49585/18?u=paulanthonywilson">here, on moving a project from Umbrella to flat</a></p>
<p>Also, from the same post,</p>
<blockquote>
<p>faster test and build times</p>
</blockquote>
<p>Having not worked on huge Umbrealla Projects this is not something I have noticed. Unfortunately there are not timings, but <a href="https://elixirforum.com/t/what-s-wrong-with-umbrella-apps/49585/22?u=paulanthonywilson">Saša Jurić says that the difference is noticeable</a>.</p>
<h3 id="there-are-better-ways-to-achieve-the-same-objectives">There are better ways to achieve the same objectives</h3>
<blockquote>
<p>A better solution is to just be unafraid to make top level namespaces in the main app (like how Phoenix creates MyApp and MyAppWeb)</p>
</blockquote>
<p><a href="https://elixirforum.com/t/what-s-wrong-with-umbrella-apps/49585/3?u=paulanthonywilson">from here</a></p>
<p>I like that.</p>
<blockquote>
<p>Tools like [Saša Jurić’s Boundary] offer a better way to tackle the “I don’t want code from this module to be called by this other module” problem in a saner way that doesn’t break most of the tools out there.</p>
</blockquote>
<p>I remember taking a look at <a href="https://hexdocs.pm/boundary/Boundary.html">Boundary</a> a while back, but did not try it for some reason. I think it was a bit early-doors at the time, and I was satisfied with Umbrellas. It seems pretty solid now, and I will definitely try it out in future.</p>
<h2 id="in-conclusion">In Conclusion</h2>
<blockquote>
<p>The man who never alters his opinion is like standing water, and breeds reptiles of the mind.</p>
</blockquote>
<p><a href="https://www.gutenberg.org/cache/epub/45315/pg45315.txt">William Blake, The Marrige of Heaven and Hell</a></p>
<p>I am sufficiently convinced to take a step back from Umbrella Projects, despite using them for many years, and try some different approaches to get the same separation of concerns and dependency directionality. I am grateful to the participants in the <a href="https://elixirforum.com/t/what-s-wrong-with-umbrella-apps/49585?u=paulanthonywilson">Elixir Forum thread</a> for the enlightening discussion.</p>
<h2 id="updates"><strong>Updates</strong></h2>
<ul>
<li>2023-01-20 - Slight sentence structure and narrative flow updates.</li>
</ul>
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>Umbrella Projects, that is. I prefer a waterproof jacket to an actual umbrella, especially here in windy Scotland. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>I think I may have made up the term <em>Flat App</em>, for apps which are not Umbrellas, but I need to call them something and it’s not going to be <em>Non-Umbrella Projects</em>, so don’t start with that. <em>Normie Apps</em> might be an ok term. (It’s that field hockey vs ice hockey thing.) <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>There’s no full video of Nordic Ruby talks that I can find. There’s just <a href="https://vimeo.com/27142345">this flavour of the conference video</a>. Tom appears around at 2`6”. There’s also a brief extract of one of the best conference talks I’ve seen, Chad Fowler on “Legacy Systems” at 1’13. A younger, thinner, version of me makes a brief appearance at 5’33”. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>Which was (is?) sorely lacking in Ruby on Rails. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p>I wouldn’t <a href="#fnref:5" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:6" role="doc-endnote">
<p>The rest of that post, and contributions from the author, does contain some great arguments; this is more of an aside, I think. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:7" role="doc-endnote">
<p>Never, apart from some experimental Nerves work with different interacting components. Also a Nerves + Server side deployment which I kept in a single Umbrella, but probably would not do again. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Paul WilsonI got into programming Elixir somewhere around 2013. (I’m not great with dates but I had a check of some projects on Github). Since I found out about them, every project I have initiated has been an Umbrella Project; for the past few years, though, I do seem to be the only person who likes Umbrellas1. Umbrella Projects, that is. I prefer a waterproof jacket to an actual umbrella, especially here in windy Scotland. ↩Death, Children, and OTP2021-06-28T20:35:36+00:002021-06-28T20:35:36+00:00/elixir/otp/2021/06/28/death-children-and-otp<h1 id="death-children-and-otp">Death, children, and OTP</h1>
<h2 id="summary">Summary</h2>
<p>Previously in this (what has become a) series of posts we have looked at <a href="https://furlough.merecomplexities.com/elixir/otp/2021/05/31/the-many-and-varied-ways-to-kill-an-otp-process.html">how an OTP process
behaves when we try and kill it</a>
and <a href="https://furlough.merecomplexities.com/elixir/otp/2021/06/08/what-happens-when-a-linked-process-dies.html">what happens to an OTP process when a linked process dies</a>.</p>
<p>In an <a href="https://elixirforum.com/t/elixir-blog-post-the-many-and-varied-ways-to-kill-an-otp-process/40135">Elixir Form thread</a>
a forum member, <a href="https://elixirforum.com/u/the_wildgoose/">The Wild Goose</a>, pointed out that there was special
behaviour when a the exit signal is received from a processes parent. I was sceptical of this, finding no
reference to parent / child relationship in the
<a href="http://erlang.org/doc/reference_manual/processes.html#signals">Erlang process documentation</a>.</p>
<p>I was wrong to be sceptical. While not being
part of Erlang, there is parent/child behaviour specified in the
<a href="http://erlang.org/doc/design_principles/spec_proc.html">OTP documentation</a>: when a linked parent dies then
the exit signal becomes untrappable in OTP conformant processes, such as a GenServer.</p>
<p>Here’s a summary of the behaviour.</p>
<table>
<thead>
<tr>
<th>Trapping exits?</th>
<th>Diffent to non parent/child behaviour</th>
<th>Reason for linked parent exit</th>
<th>Exit message received?</th>
<th>Exits?</th>
<th>terminate/2 callback?</th>
</tr>
</thead>
<tbody>
<tr>
<td>no</td>
<td>no</td>
<td><code class="language-plaintext highlighter-rouge">:normal</code></td>
<td>no</td>
<td>no</td>
<td>no</td>
</tr>
<tr>
<td>no</td>
<td>no</td>
<td>any reason other than <code class="language-plaintext highlighter-rouge">:normal</code></td>
<td>no</td>
<td>yes</td>
<td>no</td>
</tr>
<tr>
<td>yes</td>
<td>yes</td>
<td>any reason other than <code class="language-plaintext highlighter-rouge">:normal</code></td>
<td>no</td>
<td>yes</td>
<td>yes</td>
</tr>
<tr>
<td>yes</td>
<td>yes</td>
<td><code class="language-plaintext highlighter-rouge">:normal</code></td>
<td>no</td>
<td>yes</td>
<td>yes</td>
</tr>
</tbody>
</table>
<p>So indeed, exits are not trapped by OTP processes, eg GenServer, when the processes parent dies - parent being
the process that has created this process.</p>
<p>What raised my left eyebrow was that if the parent process exits
with a <code class="language-plaintext highlighter-rouge">:normal</code> reason then this does not affect the child unless the child is trapping exits; when trapping
exits a parent terminating with <code class="language-plaintext highlighter-rouge">:normal</code> also causes the child to die.</p>
<p>The <a href="https://github.com/elixir-nx/livebook">LiveBook</a> page of this post is
<a href="https://github.com/paulanthonywilson/furlough/blob/master/_posts/livebook/death-and-children.livemd">here</a> and
you can follow the instructions in
<a href="https://furlough.merecomplexities.com/elixir/otp/2021/05/31/the-many-and-varied-ways-to-kill-an-otp-process.html#executing-as-livebook">this post</a>
to execute.</p>
<h2 id="code">Code</h2>
<p>Here is a GenServer that we are going to use to investigate this parent / child exit behaviour.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">Life</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">GenServer</span>
<span class="k">def</span> <span class="n">trap_exits</span><span class="p">(</span><span class="n">server</span><span class="p">)</span> <span class="k">do</span>
<span class="ss">:ok</span> <span class="o">=</span> <span class="no">GenServer</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="n">server</span><span class="p">,</span> <span class="ss">:trap_exits</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">make_parent_and_child</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">parent</span><span class="p">}</span> <span class="o">=</span> <span class="no">GenServer</span><span class="o">.</span><span class="n">start</span><span class="p">(</span><span class="bp">__MODULE__</span><span class="p">,</span> <span class="ss">:parent</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">child</span><span class="p">}</span> <span class="o">=</span> <span class="no">GenServer</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="n">parent</span><span class="p">,</span> <span class="ss">:make_child</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">parent</span><span class="p">,</span> <span class="n">child</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">make_parent_and_linked_child</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">parent</span><span class="p">}</span> <span class="o">=</span> <span class="no">GenServer</span><span class="o">.</span><span class="n">start</span><span class="p">(</span><span class="bp">__MODULE__</span><span class="p">,</span> <span class="ss">:parent</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">child</span><span class="p">}</span> <span class="o">=</span> <span class="no">GenServer</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="n">parent</span><span class="p">,</span> <span class="ss">:make_linked_child</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">parent</span><span class="p">,</span> <span class="n">child</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">alive_after_wait_for_death?</span><span class="p">(</span><span class="n">pid</span><span class="p">,</span> <span class="n">count</span> <span class="p">\\</span> <span class="mi">50</span><span class="p">)</span>
<span class="k">def</span> <span class="n">alive_after_wait_for_death?</span><span class="p">(</span><span class="n">pid</span><span class="p">,</span> <span class="mi">0</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="no">Process</span><span class="o">.</span><span class="n">alive?</span><span class="p">(</span><span class="n">pid</span><span class="p">)</span>
<span class="k">def</span> <span class="n">alive_after_wait_for_death?</span><span class="p">(</span><span class="n">pid</span><span class="p">,</span> <span class="n">count</span><span class="p">)</span> <span class="k">do</span>
<span class="k">case</span> <span class="no">Process</span><span class="o">.</span><span class="n">alive?</span><span class="p">(</span><span class="n">pid</span><span class="p">)</span> <span class="k">do</span>
<span class="no">true</span> <span class="o">-></span>
<span class="ss">:timer</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="n">alive_after_wait_for_death?</span><span class="p">(</span><span class="n">pid</span><span class="p">,</span> <span class="n">count</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">_</span> <span class="o">-></span>
<span class="no">false</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">init</span><span class="p">(</span><span class="n">tag</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="p">%{</span><span class="ss">tag:</span> <span class="n">tag</span><span class="p">}}</span>
<span class="k">def</span> <span class="n">handle_call</span><span class="p">(</span><span class="ss">:trap_exits</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">s</span><span class="p">)</span> <span class="k">do</span>
<span class="no">Process</span><span class="o">.</span><span class="n">flag</span><span class="p">(</span><span class="ss">:trap_exit</span><span class="p">,</span> <span class="no">true</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:reply</span><span class="p">,</span> <span class="ss">:ok</span><span class="p">,</span> <span class="n">s</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">handle_call</span><span class="p">(</span><span class="ss">:make_child</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">s</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:reply</span><span class="p">,</span> <span class="no">GenServer</span><span class="o">.</span><span class="n">start</span><span class="p">(</span><span class="bp">__MODULE__</span><span class="p">,</span> <span class="ss">:child</span><span class="p">),</span> <span class="n">s</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">handle_call</span><span class="p">(</span><span class="ss">:make_linked_child</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">s</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:reply</span><span class="p">,</span> <span class="no">GenServer</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="bp">__MODULE__</span><span class="p">,</span> <span class="ss">:child</span><span class="p">),</span> <span class="n">s</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">handle_info</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="p">%{</span><span class="ss">tag:</span> <span class="n">tag</span><span class="p">}</span> <span class="o">=</span> <span class="n">s</span><span class="p">)</span> <span class="k">do</span>
<span class="no">IO</span><span class="o">.</span><span class="n">inspect</span><span class="p">({</span><span class="n">tag</span><span class="p">,</span> <span class="n">self</span><span class="p">(),</span> <span class="n">event</span><span class="p">},</span> <span class="ss">label:</span> <span class="ss">:handle_info</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:noreply</span><span class="p">,</span> <span class="n">s</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">terminate</span><span class="p">(</span><span class="n">reason</span><span class="p">,</span> <span class="p">%{</span><span class="ss">tag:</span> <span class="n">tag</span><span class="p">})</span> <span class="k">do</span>
<span class="no">IO</span><span class="o">.</span><span class="n">inspect</span><span class="p">({</span><span class="n">tag</span><span class="p">,</span> <span class="n">self</span><span class="p">(),</span> <span class="n">reason</span><span class="p">},</span> <span class="ss">label:</span> <span class="ss">:terminate</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<h2 id="effect-of-killing-a-parent-on-its-unlinked-child">Effect of killing a parent on its unlinked child</h2>
<p>Just to get it out of the way let’s look at what happens to an unlinked child when the parent exits.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">parent</span><span class="p">,</span> <span class="n">child</span><span class="p">}</span> <span class="o">=</span> <span class="no">Life</span><span class="o">.</span><span class="n">make_parent_and_child</span><span class="p">()</span>
<span class="no">Process</span><span class="o">.</span><span class="k">exit</span><span class="p">(</span><span class="n">parent</span><span class="p">,</span> <span class="ss">:kill</span><span class="p">)</span>
<span class="no">true</span> <span class="o">=</span> <span class="no">Life</span><span class="o">.</span><span class="n">alive_after_wait_for_death?</span><span class="p">(</span><span class="n">child</span><span class="p">)</span>
</code></pre></div></div>
<p>As we would expect, when there is no link the parent’s exit has no impact on the child.</p>
<h2 id="effect-of-killing-a-parent-on-a-linked-child">Effect of killing a parent on a linked child</h2>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">parent</span><span class="p">,</span> <span class="n">child</span><span class="p">}</span> <span class="o">=</span> <span class="no">Life</span><span class="o">.</span><span class="n">make_parent_and_linked_child</span><span class="p">()</span>
<span class="no">Process</span><span class="o">.</span><span class="k">exit</span><span class="p">(</span><span class="n">parent</span><span class="p">,</span> <span class="ss">:kill</span><span class="p">)</span>
<span class="no">false</span> <span class="o">=</span> <span class="no">Life</span><span class="o">.</span><span class="n">alive_after_wait_for_death?</span><span class="p">(</span><span class="n">child</span><span class="p">)</span>
</code></pre></div></div>
<p>Of course, if we kill a linked parent then the child will also die without calling <code class="language-plaintext highlighter-rouge">terminate/2</code>,
just like killing any other linked process. No suprise here.</p>
<h2 id="parent-exits-with-a-reason-other-than-normal">Parent exits with a reason other than :normal</h2>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">parent</span><span class="p">,</span> <span class="n">child</span><span class="p">}</span> <span class="o">=</span> <span class="no">Life</span><span class="o">.</span><span class="n">make_parent_and_linked_child</span><span class="p">()</span>
<span class="no">GenServer</span><span class="o">.</span><span class="n">stop</span><span class="p">(</span><span class="n">parent</span><span class="p">,</span> <span class="ss">:some_reason</span><span class="p">)</span>
<span class="no">false</span> <span class="o">=</span> <span class="no">Life</span><span class="o">.</span><span class="n">alive_after_wait_for_death?</span><span class="p">(</span><span class="n">child</span><span class="p">)</span>
</code></pre></div></div>
<p>When a linked process exits, a child that is not trapping exits will also die without a callback to
<code class="language-plaintext highlighter-rouge">terminate/2</code>, regardless of whether there is a parent/child relationship.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">parent</span><span class="p">,</span> <span class="n">child</span><span class="p">}</span> <span class="o">=</span> <span class="no">Life</span><span class="o">.</span><span class="n">make_parent_and_linked_child</span><span class="p">()</span>
<span class="no">Life</span><span class="o">.</span><span class="n">trap_exits</span><span class="p">(</span><span class="n">child</span><span class="p">)</span>
<span class="no">GenServer</span><span class="o">.</span><span class="n">stop</span><span class="p">(</span><span class="n">parent</span><span class="p">,</span> <span class="ss">:some_reason</span><span class="p">)</span>
<span class="no">false</span> <span class="o">=</span> <span class="no">Life</span><span class="o">.</span><span class="n">alive_after_wait_for_death?</span><span class="p">(</span><span class="n">child</span><span class="p">)</span>
</code></pre></div></div>
<p>In contrast to a non parent / child linked process exit, if the child is trapping exits and the parent dies,
the exit is not trapped and the child will still die. The <code class="language-plaintext highlighter-rouge">terminate/2</code> callback is called (if supplied).</p>
<h2 id="how-about-the-effect-on-a-linked-child-when-a-parent-exits-normally">How about the effect on a linked child when a parent exits normally?</h2>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">parent</span><span class="p">,</span> <span class="n">child</span><span class="p">}</span> <span class="o">=</span> <span class="no">Life</span><span class="o">.</span><span class="n">make_parent_and_linked_child</span><span class="p">()</span>
<span class="no">GenServer</span><span class="o">.</span><span class="n">stop</span><span class="p">(</span><span class="n">parent</span><span class="p">,</span> <span class="ss">:normal</span><span class="p">)</span>
<span class="no">true</span> <span class="o">=</span> <span class="no">Life</span><span class="o">.</span><span class="n">alive_after_wait_for_death?</span><span class="p">(</span><span class="n">child</span><span class="p">)</span>
</code></pre></div></div>
<p>A process exiting with a <code class="language-plaintext highlighter-rouge">:normal</code> reason, does not cause any linked processes to exit regardless of a parent/child
relationship.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">parent</span><span class="p">,</span> <span class="n">child</span><span class="p">}</span> <span class="o">=</span> <span class="no">Life</span><span class="o">.</span><span class="n">make_parent_and_linked_child</span><span class="p">()</span>
<span class="no">Life</span><span class="o">.</span><span class="n">trap_exits</span><span class="p">(</span><span class="n">child</span><span class="p">)</span>
<span class="no">GenServer</span><span class="o">.</span><span class="n">stop</span><span class="p">(</span><span class="n">parent</span><span class="p">,</span> <span class="ss">:normal</span><span class="p">)</span>
<span class="no">false</span> <span class="o">=</span> <span class="no">Life</span><span class="o">.</span><span class="n">alive_after_wait_for_death?</span><span class="p">(</span><span class="n">child</span><span class="p">)</span>
</code></pre></div></div>
<p>If a child is trapping exits and its parent dies with a <code class="language-plaintext highlighter-rouge">:normal</code> reason then the child will also exit with the
same reason and a callback to <code class="language-plaintext highlighter-rouge">terminate/2</code>. It’s all a bit
<a href="https://en.wikipedia.org/wiki/Laius">Greek and tragic</a>: only by doing the thing that you might expect would
prevent the child’s death (trapping exits), does its death come about.</p>
<h2 id="this-series">This series</h2>
<p>Starting out looking at exit signals and OTP process death has turned into a small series of posts, including this one. These are:</p>
<ul>
<li>
<p><a href="/elixir/otp/2021/05/31/the-many-and-varied-ways-to-kill-an-otp-process.html">The many and varied ways to kill an OTP Process</a>: investigation of different ways to cause (or fail to cause) a process to exit.</p>
</li>
<li>
<p><a href="/elixir/otp/2021/06/08/what-happens-when-a-linked-process-dies.html">What happens when a linked process dies</a>: the impact of a process exiting on processes that are linked to it, excluding OTP processes with a parent/child relationship.</p>
</li>
<li>
<p><a href="/elixir/otp/2021/06/28/death-children-and-otp.html">Death, Children, and OTP</a>: the impact on an OTP process when
the process that spawned it (its <em>parent</em>) exits, particularly when the child is trapping exits.</p>
</li>
</ul>
<h2 id="updates">Updates</h2>
<ul>
<li>
<p><strong>2021-06-29</strong>: included the section linking to posts in this series.</p>
</li>
<li>
<p><strong>2021-06-29</strong>: added a reference to Oedipus and Laius to make me seem erudite (though in that case it
was the child killing the parent so it’s maybe a bit misleading).</p>
</li>
</ul>Paul WilsonDeath, children, and OTP Summary Previously in this (what has become a) series of posts we have looked at how an OTP process behaves when we try and kill it and what happens to an OTP process when a linked process dies. In an Elixir Form thread a forum member, The Wild Goose, pointed out that there was special behaviour when a the exit signal is received from a processes parent. I was sceptical of this, finding no reference to parent / child relationship in the Erlang process documentation. I was wrong to be sceptical. While not being part of Erlang, there is parent/child behaviour specified in the OTP documentation: when a linked parent dies then the exit signal becomes untrappable in OTP conformant processes, such as a GenServer. Here’s a summary of the behaviour. Trapping exits? Diffent to non parent/child behaviour Reason for linked parent exit Exit message received? Exits? terminate/2 callback? no no :normal no no no no no any reason other than :normal no yes no yes yes any reason other than :normal no yes yes yes yes :normal no yes yes So indeed, exits are not trapped by OTP processes, eg GenServer, when the processes parent dies - parent being the process that has created this process. What raised my left eyebrow was that if the parent process exits with a :normal reason then this does not affect the child unless the child is trapping exits; when trapping exits a parent terminating with :normal also causes the child to die. The LiveBook page of this post is here and you can follow the instructions in this post to execute. Code Here is a GenServer that we are going to use to investigate this parent / child exit behaviour. defmodule Life do use GenServer def trap_exits(server) do :ok = GenServer.call(server, :trap_exits) end def make_parent_and_child do {:ok, parent} = GenServer.start(__MODULE__, :parent) {:ok, child} = GenServer.call(parent, :make_child) {:ok, parent, child} end def make_parent_and_linked_child do {:ok, parent} = GenServer.start(__MODULE__, :parent) {:ok, child} = GenServer.call(parent, :make_linked_child) {:ok, parent, child} end def alive_after_wait_for_death?(pid, count \\ 50) def alive_after_wait_for_death?(pid, 0), do: Process.alive?(pid) def alive_after_wait_for_death?(pid, count) do case Process.alive?(pid) do true -> :timer.sleep(1) alive_after_wait_for_death?(pid, count - 1) _ -> false end end def init(tag), do: {:ok, %{tag: tag}} def handle_call(:trap_exits, _, s) do Process.flag(:trap_exit, true) {:reply, :ok, s} end def handle_call(:make_child, _, s) do {:reply, GenServer.start(__MODULE__, :child), s} end def handle_call(:make_linked_child, _, s) do {:reply, GenServer.start_link(__MODULE__, :child), s} end def handle_info(event, %{tag: tag} = s) do IO.inspect({tag, self(), event}, label: :handle_info) {:noreply, s} end def terminate(reason, %{tag: tag}) do IO.inspect({tag, self(), reason}, label: :terminate) end end Effect of killing a parent on its unlinked child Just to get it out of the way let’s look at what happens to an unlinked child when the parent exits. {:ok, parent, child} = Life.make_parent_and_child() Process.exit(parent, :kill) true = Life.alive_after_wait_for_death?(child) As we would expect, when there is no link the parent’s exit has no impact on the child. Effect of killing a parent on a linked child {:ok, parent, child} = Life.make_parent_and_linked_child() Process.exit(parent, :kill) false = Life.alive_after_wait_for_death?(child) Of course, if we kill a linked parent then the child will also die without calling terminate/2, just like killing any other linked process. No suprise here. Parent exits with a reason other than :normal {:ok, parent, child} = Life.make_parent_and_linked_child() GenServer.stop(parent, :some_reason) false = Life.alive_after_wait_for_death?(child) When a linked process exits, a child that is not trapping exits will also die without a callback to terminate/2, regardless of whether there is a parent/child relationship. {:ok, parent, child} = Life.make_parent_and_linked_child() Life.trap_exits(child) GenServer.stop(parent, :some_reason) false = Life.alive_after_wait_for_death?(child) In contrast to a non parent / child linked process exit, if the child is trapping exits and the parent dies, the exit is not trapped and the child will still die. The terminate/2 callback is called (if supplied). How about the effect on a linked child when a parent exits normally? {:ok, parent, child} = Life.make_parent_and_linked_child() GenServer.stop(parent, :normal) true = Life.alive_after_wait_for_death?(child) A process exiting with a :normal reason, does not cause any linked processes to exit regardless of a parent/child relationship. {:ok, parent, child} = Life.make_parent_and_linked_child() Life.trap_exits(child) GenServer.stop(parent, :normal) false = Life.alive_after_wait_for_death?(child) If a child is trapping exits and its parent dies with a :normal reason then the child will also exit with the same reason and a callback to terminate/2. It’s all a bit Greek and tragic: only by doing the thing that you might expect would prevent the child’s death (trapping exits), does its death come about.I have merged in my old blog to this one2021-06-10T15:51:34+00:002021-06-10T15:51:34+00:00/not-interesting/2021/06/10/i-have-merged-in-my-old-blog-to-this-one<p>I started my first blog in about 2005, I think. That is now lost. Back in 2014 <a href="/2014/07/11/new-blog.html">I started a new one</a> which did not really stay active long because of all the <a href="https://www.twitter.com/paulanthonywils">tweeting</a>; I’ve just merged that into here and redirected my main url, <a href="http://merecomplexities.com">http://merecomplexities.com</a>.</p>
<p>I should really sort things out and</p>
<ul>
<li>get some sort of https redirection going on from <code class="language-plaintext highlighter-rouge">merecomplexities.com</code></li>
<li>make this soemething like <code class="language-plaintext highlighter-rouge">weblog.merecomp.com</code> and (https) redirect from <code class="language-plaintext highlighter-rouge">furloughlog.merecomplexities.com</code></li>
<li>make it beautiful</li>
</ul>
<p>So, the last one isn’t going to happen. I don’t want to pay $300 a year for DNSimple https redirection and I don’t trust the free ones. I
guess I may end up with a Digital Ocean droplet that just redirects.</p>
<p>(I do need to get my Let’s Encrypt SSL certificate game sorted out first, though.)</p>Paul WilsonI started my first blog in about 2005, I think. That is now lost. Back in 2014 I started a new one which did not really stay active long because of all the tweeting; I’ve just merged that into here and redirected my main url, http://merecomplexities.com.What happens when a linked process dies2021-06-08T10:27:41+00:002021-06-08T10:27:41+00:00/elixir/otp/2021/06/08/what-happens-when-a-linked-process-dies<h1 id="what-happens-when-a-linked-process-dies">What happens when a linked process dies</h1>
<h2 id="summary">Summary</h2>
<p><a href="https://furlough.merecomplexities.com/elixir/otp/2021/05/31/the-many-and-varied-ways-to-kill-an-otp-process.html">Previously</a>
we looked at different ways we can kill (or attempt to kill) a process and what happens in each case. Now let’s
see what happens when a linked process dies. The tl;dr is in the table below.</p>
<table>
<thead>
<tr>
<th>Trapping exits?</th>
<th>Reason for linked process exit</th>
<th>Exit message received?</th>
<th>Exits?</th>
</tr>
</thead>
<tbody>
<tr>
<td>no</td>
<td><code class="language-plaintext highlighter-rouge">:normal</code></td>
<td>no</td>
<td>no</td>
</tr>
<tr>
<td>no</td>
<td>any reason other than <code class="language-plaintext highlighter-rouge">:normal</code></td>
<td>no</td>
<td>yes</td>
</tr>
<tr>
<td>yes</td>
<td>any reason including <code class="language-plaintext highlighter-rouge">:normal</code></td>
<td>yes</td>
<td>no</td>
</tr>
</tbody>
</table>
<p>Note that the behaviour describes what happens when the exiting linked process is _not_the parent process. We
look at that in a
<a href="https://furlough.merecomplexities.com/elixir/otp/2021/06/28/death-children-and-otp.html">subsequent post</a>.</p>
<p>This page can be downloaded as a LiveBook for execution. The raw markdown is <a href="https://raw.githubusercontent.com/paulanthonywilson/furlough/master/_posts/livebook/linked-process-death.livemd">here</a>,
or you can follow the <a href="https://furlough.merecomplexities.com/elixir/otp/2021/05/31/the-many-and-varied-ways-to-kill-an-otp-process.html#executing-as-livebook">previous instructions</a>
for cloning the blog and executing the LiveBook pages.</p>
<p>When the final value of a snippet is significant I have matched against it to make the output clearer when
reading the blog rather than executing the page. eg</p>
<!-- livebook:{"force_markdown":true} -->
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">true</span> <span class="o">=</span> <span class="no">Linky</span><span class="o">.</span><span class="n">alive_after_wait_for_death?</span><span class="p">(</span><span class="n">l1</span><span class="p">)</span>
</code></pre></div></div>
<h2 id="a-genserver-for-the-experiments">A GenServer for the experiments</h2>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">Linky</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">GenServer</span>
<span class="k">def</span> <span class="n">start</span> <span class="k">do</span>
<span class="no">GenServer</span><span class="o">.</span><span class="n">start</span><span class="p">(</span><span class="bp">__MODULE__</span><span class="p">,</span> <span class="p">{})</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">init</span><span class="p">(</span><span class="n">_</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="p">%{}}</span>
<span class="k">def</span> <span class="n">link</span><span class="p">(</span><span class="n">server</span><span class="p">,</span> <span class="n">other</span><span class="p">)</span> <span class="k">do</span>
<span class="ss">:ok</span> <span class="o">=</span> <span class="no">GenServer</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="n">server</span><span class="p">,</span> <span class="p">{</span><span class="ss">:link</span><span class="p">,</span> <span class="n">other</span><span class="p">})</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">trap_exits</span><span class="p">(</span><span class="n">server</span><span class="p">)</span> <span class="k">do</span>
<span class="ss">:ok</span> <span class="o">=</span> <span class="no">GenServer</span><span class="o">.</span><span class="n">call</span><span class="p">(</span><span class="n">server</span><span class="p">,</span> <span class="ss">:trap_exits</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">alive_after_wait_for_death?</span><span class="p">(</span><span class="n">pid</span><span class="p">,</span> <span class="n">count</span> <span class="p">\\</span> <span class="mi">50</span><span class="p">)</span>
<span class="k">def</span> <span class="n">alive_after_wait_for_death?</span><span class="p">(</span><span class="n">pid</span><span class="p">,</span> <span class="mi">0</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="no">Process</span><span class="o">.</span><span class="n">alive?</span><span class="p">(</span><span class="n">pid</span><span class="p">)</span>
<span class="k">def</span> <span class="n">alive_after_wait_for_death?</span><span class="p">(</span><span class="n">pid</span><span class="p">,</span> <span class="n">count</span><span class="p">)</span> <span class="k">do</span>
<span class="k">case</span> <span class="no">Process</span><span class="o">.</span><span class="n">alive?</span><span class="p">(</span><span class="n">pid</span><span class="p">)</span> <span class="k">do</span>
<span class="no">true</span> <span class="o">-></span>
<span class="ss">:timer</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="n">alive_after_wait_for_death?</span><span class="p">(</span><span class="n">pid</span><span class="p">,</span> <span class="n">count</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">_</span> <span class="o">-></span>
<span class="no">false</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">handle_call</span><span class="p">({</span><span class="ss">:link</span><span class="p">,</span> <span class="n">other</span><span class="p">},</span> <span class="n">_</span><span class="p">,</span> <span class="n">s</span><span class="p">)</span> <span class="k">do</span>
<span class="no">Process</span><span class="o">.</span><span class="n">link</span><span class="p">(</span><span class="n">other</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:reply</span><span class="p">,</span> <span class="ss">:ok</span><span class="p">,</span> <span class="n">s</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">handle_call</span><span class="p">(</span><span class="ss">:trap_exits</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">s</span><span class="p">)</span> <span class="k">do</span>
<span class="no">Process</span><span class="o">.</span><span class="n">flag</span><span class="p">(</span><span class="ss">:trap_exit</span><span class="p">,</span> <span class="no">true</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:reply</span><span class="p">,</span> <span class="ss">:ok</span><span class="p">,</span> <span class="n">s</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">handle_info</span><span class="p">(</span><span class="n">event</span><span class="p">,</span> <span class="n">s</span><span class="p">)</span> <span class="k">do</span>
<span class="no">IO</span><span class="o">.</span><span class="n">inspect</span><span class="p">({</span><span class="n">self</span><span class="p">(),</span> <span class="n">event</span><span class="p">},</span> <span class="ss">label:</span> <span class="ss">:handle_info</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:noreply</span><span class="p">,</span> <span class="n">s</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">terminate</span><span class="p">(</span><span class="n">reason</span><span class="p">,</span> <span class="n">_s</span><span class="p">)</span> <span class="k">do</span>
<span class="no">IO</span><span class="o">.</span><span class="n">inspect</span><span class="p">({</span><span class="n">self</span><span class="p">(),</span> <span class="n">reason</span><span class="p">},</span> <span class="ss">label:</span> <span class="ss">:terminate</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>The GenServer above is used in the illustrative code that follows.</p>
<h2 id="linked-process-exits-with-normal-reason">Linked process exits with :normal reason</h2>
<p>I’ve read in more than one place, that linking two processes ties their lifecycle together, so that one dies
the other pops off too. This is not the full story.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">l1</span><span class="p">}</span> <span class="o">=</span> <span class="no">Linky</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">l2</span><span class="p">}</span> <span class="o">=</span> <span class="no">Linky</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="no">Linky</span><span class="o">.</span><span class="n">link</span><span class="p">(</span><span class="n">l1</span><span class="p">,</span> <span class="n">l2</span><span class="p">)</span>
<span class="no">GenServer</span><span class="o">.</span><span class="n">stop</span><span class="p">(</span><span class="n">l2</span><span class="p">,</span> <span class="ss">:normal</span><span class="p">)</span>
<span class="no">true</span> <span class="o">=</span> <span class="no">Linky</span><span class="o">.</span><span class="n">alive_after_wait_for_death?</span><span class="p">(</span><span class="n">l1</span><span class="p">)</span>
</code></pre></div></div>
<p>When we execute the above code, we discover that when a linked process exits with a <code class="language-plaintext highlighter-rouge">:normal</code> reason then the
other process does not die.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">l1</span><span class="p">}</span> <span class="o">=</span> <span class="no">Linky</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">l2</span><span class="p">}</span> <span class="o">=</span> <span class="no">Linky</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="no">Linky</span><span class="o">.</span><span class="n">link</span><span class="p">(</span><span class="n">l1</span><span class="p">,</span> <span class="n">l2</span><span class="p">)</span>
<span class="no">Linky</span><span class="o">.</span><span class="n">trap_exits</span><span class="p">(</span><span class="n">l1</span><span class="p">)</span>
<span class="no">GenServer</span><span class="o">.</span><span class="n">stop</span><span class="p">(</span><span class="n">l2</span><span class="p">,</span> <span class="ss">:normal</span><span class="p">)</span>
<span class="no">true</span> <span class="o">=</span> <span class="no">Linky</span><span class="o">.</span><span class="n">alive_after_wait_for_death?</span><span class="p">(</span><span class="n">l1</span><span class="p">)</span>
</code></pre></div></div>
<p>We still get a message from a linked process’s <code class="language-plaintext highlighter-rouge">:normal</code> exit (eg <code class="language-plaintext highlighter-rouge">{:EXIT, #PID<0.136.0>, :normal}</code>) when we are
trapping exits. In fact, with one exception, given linked procceses l1 and l2, when l1 experiences l2’s exit
exactly as if l2 had called <code class="language-plaintext highlighter-rouge">Process.exit/2</code> on <code class="language-plaintext highlighter-rouge">l1</code> with its exit reason.</p>
<h2 id="linked-processes-that-exit-by-a-kill">Linked processes that exit by a :kill</h2>
<p>Remember that <code class="language-plaintext highlighter-rouge">:kill</code> is an untrappable exit when calling <code class="language-plaintext highlighter-rouge">Process.exit/2</code>? Its untrappability does not cascade
to linked processes.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">l1</span><span class="p">}</span> <span class="o">=</span> <span class="no">Linky</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">l2</span><span class="p">}</span> <span class="o">=</span> <span class="no">Linky</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="no">Linky</span><span class="o">.</span><span class="n">link</span><span class="p">(</span><span class="n">l1</span><span class="p">,</span> <span class="n">l2</span><span class="p">)</span>
<span class="no">Linky</span><span class="o">.</span><span class="n">trap_exits</span><span class="p">(</span><span class="n">l1</span><span class="p">)</span>
<span class="no">Process</span><span class="o">.</span><span class="k">exit</span><span class="p">(</span><span class="n">l2</span><span class="p">,</span> <span class="ss">:kill</span><span class="p">)</span>
<span class="no">true</span> <span class="o">=</span> <span class="no">Linky</span><span class="o">.</span><span class="n">alive_after_wait_for_death?</span><span class="p">(</span><span class="n">l1</span><span class="p">)</span>
</code></pre></div></div>
<p>Interestingly the reason received by the linked process is <code class="language-plaintext highlighter-rouge">:killed</code> not <code class="language-plaintext highlighter-rouge">:kill</code>, which would of course be
trappable if sent via <code class="language-plaintext highlighter-rouge">Process.exit/2</code>. That is one explanation for <code class="language-plaintext highlighter-rouge">:kill</code>s not cascasding through linked
processes.</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">l1</span><span class="p">}</span> <span class="o">=</span> <span class="no">Linky</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">l2</span><span class="p">}</span> <span class="o">=</span> <span class="no">Linky</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="no">Linky</span><span class="o">.</span><span class="n">link</span><span class="p">(</span><span class="n">l1</span><span class="p">,</span> <span class="n">l2</span><span class="p">)</span>
<span class="no">Linky</span><span class="o">.</span><span class="n">trap_exits</span><span class="p">(</span><span class="n">l1</span><span class="p">)</span>
<span class="no">GenServer</span><span class="o">.</span><span class="n">stop</span><span class="p">(</span><span class="n">l2</span><span class="p">,</span> <span class="ss">:kill</span><span class="p">)</span>
<span class="no">true</span> <span class="o">=</span> <span class="no">Linky</span><span class="o">.</span><span class="n">alive_after_wait_for_death?</span><span class="p">(</span><span class="n">l1</span><span class="p">)</span>
</code></pre></div></div>
<p>It would be an explanation for <code class="language-plaintext highlighter-rouge">kill</code>s not cascading, except it is perfectly possible, as above, for a process to
exit with a reason<code class="language-plaintext highlighter-rouge">:kill</code>, and have the signal <code class="language-plaintext highlighter-rouge">:kill</code> trapped by a linked process.
(Possible, but it would be an curious thing to actually do in production code.)</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">l1</span><span class="p">}</span> <span class="o">=</span> <span class="no">Linky</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">l2</span><span class="p">}</span> <span class="o">=</span> <span class="no">Linky</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="no">Linky</span><span class="o">.</span><span class="n">link</span><span class="p">(</span><span class="n">l1</span><span class="p">,</span> <span class="n">l2</span><span class="p">)</span>
<span class="no">Process</span><span class="o">.</span><span class="k">exit</span><span class="p">(</span><span class="n">l2</span><span class="p">,</span> <span class="ss">:kill</span><span class="p">)</span>
<span class="no">false</span> <span class="o">=</span> <span class="no">Linky</span><span class="o">.</span><span class="n">alive_after_wait_for_death?</span><span class="p">(</span><span class="n">l1</span><span class="p">)</span>
</code></pre></div></div>
<p>Of course if we are not trapping exits and a linked process is kiled, then the other processes also dies.</p>
<h2 id="linked-processes-exiting-with-other-reasons">Linked processes exiting with other reasons</h2>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">l1</span><span class="p">}</span> <span class="o">=</span> <span class="no">Linky</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">l2</span><span class="p">}</span> <span class="o">=</span> <span class="no">Linky</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="no">Linky</span><span class="o">.</span><span class="n">link</span><span class="p">(</span><span class="n">l1</span><span class="p">,</span> <span class="n">l2</span><span class="p">)</span>
<span class="no">GenServer</span><span class="o">.</span><span class="n">stop</span><span class="p">(</span><span class="n">l2</span><span class="p">,</span> <span class="ss">:whatever</span><span class="p">)</span>
<span class="no">false</span> <span class="o">=</span> <span class="no">Linky</span><span class="o">.</span><span class="n">alive_after_wait_for_death?</span><span class="p">(</span><span class="n">l1</span><span class="p">)</span>
</code></pre></div></div>
<p>For completeness, the above code shows that if you are not trapping exits then linked processes will exit when
the other exits, as long as the reason is not <code class="language-plaintext highlighter-rouge">:normal</code>.</p>
<h2 id="linked-non-genserver--otp-processes-that-simply-exit">Linked (non GenServer / OTP) processes that simply exit</h2>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">l1</span><span class="p">}</span> <span class="o">=</span> <span class="no">Linky</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">l2</span><span class="p">}</span> <span class="o">=</span> <span class="no">Linky</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">spawny</span> <span class="o">=</span>
<span class="n">spawn</span><span class="p">(</span><span class="k">fn</span> <span class="o">-></span>
<span class="k">receive</span> <span class="k">do</span>
<span class="ss">:bye</span> <span class="o">-></span>
<span class="no">IO</span><span class="o">.</span><span class="n">puts</span><span class="p">(</span><span class="s2">"Goodbye sweet world 😿"</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span><span class="p">)</span>
<span class="no">Linky</span><span class="o">.</span><span class="n">link</span><span class="p">(</span><span class="n">l1</span><span class="p">,</span> <span class="n">spawny</span><span class="p">)</span>
<span class="no">Linky</span><span class="o">.</span><span class="n">link</span><span class="p">(</span><span class="n">l2</span><span class="p">,</span> <span class="n">spawny</span><span class="p">)</span>
<span class="no">Linky</span><span class="o">.</span><span class="n">trap_exits</span><span class="p">(</span><span class="n">l1</span><span class="p">)</span>
<span class="n">send</span><span class="p">(</span><span class="n">spawny</span><span class="p">,</span> <span class="ss">:bye</span><span class="p">)</span>
<span class="p">[</span><span class="no">true</span><span class="p">,</span> <span class="no">true</span><span class="p">]</span> <span class="o">=</span> <span class="n">for</span> <span class="n">pid</span> <span class="o"><-</span> <span class="p">[</span><span class="n">l1</span><span class="p">,</span> <span class="n">l2</span><span class="p">],</span> <span class="k">do</span><span class="p">:</span> <span class="no">Linky</span><span class="o">.</span><span class="n">alive_after_wait_for_death?</span><span class="p">(</span><span class="n">pid</span><span class="p">)</span>
</code></pre></div></div>
<p>When any process’s function returns normally, then the process exits with the <code class="language-plaintext highlighter-rouge">:normal</code> reason. Linked processes
behave accordingly: they receive a <code class="language-plaintext highlighter-rouge">:normal</code> exit notification from the Linked process if they are trapping exits;
even if they are not trapping exits, they do not exit. This is why it’s safe, for instance, to start a task with
<code class="language-plaintext highlighter-rouge">Task.start_link/1</code> or <code class="language-plaintext highlighter-rouge">spawn_link/1</code>.</p>
<h2 id="a-silly-mistake-i-made-that-resulted-in-a-process-leak">A silly mistake I made that resulted in a process leak</h2>
<p>For <a href="https://furlough.merecomplexities.com/log/elixir/liveview/2020/06/18/memorable-password-generation-with-liveview.html#hand-rolled-debouncing">reasons</a>
I created a helper process for a <a href="https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html">LiveView</a> page in
the <a href="https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html#c:mount/3"><code class="language-plaintext highlighter-rouge">mount/3</code> callback</a>. I wrote something like</p>
<!-- livebook:{"force_markdown":true} -->
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">mount</span><span class="p">(</span><span class="n">_params</span><span class="p">,</span> <span class="n">_session</span><span class="p">,</span> <span class="n">socket</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">debouncer</span><span class="p">}</span> <span class="o">=</span> <span class="no">Debouncer</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="n">self</span><span class="p">(),</span> <span class="mi">500</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:noreply</span><span class="p">,</span> <span class="n">assign</span><span class="p">(</span><span class="n">socket</span><span class="p">,</span> <span class="ss">debouncer:</span> <span class="n">debouncer</span><span class="p">)}</span>
<span class="k">end</span>
</code></pre></div></div>
<p>You can see the problem here, right? Yes, <code class="language-plaintext highlighter-rouge">mount/3</code> is called twice: once when the initial html is rendered a
and again when the socket connects, so an extra <code class="language-plaintext highlighter-rouge">Debouncer</code> is started.</p>
<p>This would be a waste of cpu cycles, but
no more, except that the initial rendering call is initiated by a <a href="https://github.com/ninenines/cowboy/blob/e12d7bbe2151ee727d4cd63eb5df649da9b9effa/src/cowboy_stream_h.erl">Cowboy process</a>
that exits with a <code class="language-plaintext highlighter-rouge">:normal</code> reason. The <code class="language-plaintext highlighter-rouge">Debouncer</code> process does not exit, but stays orphaned and taking up
resources.</p>
<!-- livebook:{"force_markdown":true} -->
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">mount</span><span class="p">(</span><span class="n">_params</span><span class="p">,</span> <span class="n">_session</span><span class="p">,</span> <span class="n">socket</span><span class="p">)</span> <span class="k">do</span>
<span class="k">if</span> <span class="n">connected?</span><span class="p">(</span><span class="n">socket</span><span class="p">)</span> <span class="k">do</span>
<span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">debouncer</span><span class="p">}</span> <span class="o">=</span> <span class="no">Debouncer</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="n">self</span><span class="p">(),</span> <span class="mi">500</span><span class="p">)</span>
<span class="p">{</span><span class="ss">:noreply</span><span class="p">,</span> <span class="n">assign</span><span class="p">(</span><span class="n">socket</span><span class="p">,</span> <span class="ss">debouncer:</span> <span class="n">debouncer</span><span class="p">)}</span>
<span class="k">else</span>
<span class="p">{</span><span class="ss">:noreply</span><span class="p">,</span> <span class="n">socket</span><span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>The above <code class="language-plaintext highlighter-rouge">mount/3</code> does not leak processes, as the LiveView’s <a href="https://github.com/phoenixframework/phoenix_live_view/blob/0925b0b434f07b8ea17cfb5e219b80bb162026aa/lib/phoenix_live_view/channel.ex">socket process</a>
exits with <code class="language-plaintext highlighter-rouge">{:shutdown, reason}</code>, eg <code class="language-plaintext highlighter-rouge">{:shutdown, :closed}</code>, causing the linked <code class="language-plaintext highlighter-rouge">Debouncer</code> to also exit. It
would be even safer, and proof against LiveView changes, to trap exits in Debouncer and voluntarily exit with a
<code class="language-plaintext highlighter-rouge">:stop</code> return on the callback; I may do just that.</p>
<p>I thought about adding this story at the beginning of the post, but I thought it might make it like one of those
cooking articles that people complain about - the ones where you have to scroll though paragraphs of prose
before getting to the actual recipe. (Also it doesn’t reflect well on me and people might get bored before reading
this far.)</p>
<h2 id="this-series">This series</h2>
<p>Starting out looking at exit signals and OTP process death has turned into a small series of posts, including this one. These are:</p>
<ul>
<li>
<p><a href="/elixir/otp/2021/05/31/the-many-and-varied-ways-to-kill-an-otp-process.html">The many and varied ways to kill an OTP Process</a>: investigation of different ways to cause (or fail to cause) a process to exit.</p>
</li>
<li>
<p><a href="/elixir/otp/2021/06/08/what-happens-when-a-linked-process-dies.html">What happens when a linked process dies</a>: the impact of a process exiting on processes that are linked to it, excluding OTP processes with a parent/child relationship.</p>
</li>
<li>
<p><a href="/elixir/otp/2021/06/28/death-children-and-otp.html">Death, Children, and OTP</a>: the impact on an OTP process when
the process that spawned it (its <em>parent</em>) exits, particularly when the child is trapping exits.</p>
</li>
</ul>
<h2 id="updates">Updates</h2>
<ul>
<li>
<p><strong>2020-06-28</strong> Added check to code to show that a process trapping exits does not die when a linked process dies with <code class="language-plaintext highlighter-rouge">:normal</code>, just like when not trapping exits.</p>
</li>
<li>
<p><strong>2021-06-29</strong>: included the section linking to posts in this series.</p>
</li>
<li>
<p><strong>2021-06-29</strong>: added a note that this post does not look at linked processes with a parent/child relationship.</p>
</li>
</ul>Paul WilsonWhat happens when a linked process dies Summary Previously we looked at different ways we can kill (or attempt to kill) a process and what happens in each case. Now let’s see what happens when a linked process dies. The tl;dr is in the table below. Trapping exits? Reason for linked process exit Exit message received? Exits? no :normal no no no any reason other than :normal no yes yes any reason including :normal yes no Note that the behaviour describes what happens when the exiting linked process is _not_the parent process. We look at that in a subsequent post. This page can be downloaded as a LiveBook for execution. The raw markdown is here, or you can follow the previous instructions for cloning the blog and executing the LiveBook pages. When the final value of a snippet is significant I have matched against it to make the output clearer when reading the blog rather than executing the page. eg true = Linky.alive_after_wait_for_death?(l1) A GenServer for the experiments defmodule Linky do use GenServer def start do GenServer.start(__MODULE__, {}) end def init(_), do: {:ok, %{}} def link(server, other) do :ok = GenServer.call(server, {:link, other}) end def trap_exits(server) do :ok = GenServer.call(server, :trap_exits) end def alive_after_wait_for_death?(pid, count \\ 50) def alive_after_wait_for_death?(pid, 0), do: Process.alive?(pid) def alive_after_wait_for_death?(pid, count) do case Process.alive?(pid) do true -> :timer.sleep(1) alive_after_wait_for_death?(pid, count - 1) _ -> false end end def handle_call({:link, other}, _, s) do Process.link(other) {:reply, :ok, s} end def handle_call(:trap_exits, _, s) do Process.flag(:trap_exit, true) {:reply, :ok, s} end def handle_info(event, s) do IO.inspect({self(), event}, label: :handle_info) {:noreply, s} end def terminate(reason, _s) do IO.inspect({self(), reason}, label: :terminate) end end The GenServer above is used in the illustrative code that follows. Linked process exits with :normal reason I’ve read in more than one place, that linking two processes ties their lifecycle together, so that one dies the other pops off too. This is not the full story. {:ok, l1} = Linky.start() {:ok, l2} = Linky.start() Linky.link(l1, l2) GenServer.stop(l2, :normal) true = Linky.alive_after_wait_for_death?(l1) When we execute the above code, we discover that when a linked process exits with a :normal reason then the other process does not die. {:ok, l1} = Linky.start() {:ok, l2} = Linky.start() Linky.link(l1, l2) Linky.trap_exits(l1) GenServer.stop(l2, :normal) true = Linky.alive_after_wait_for_death?(l1) We still get a message from a linked process’s :normal exit (eg {:EXIT, #PID<0.136.0>, :normal}) when we are trapping exits. In fact, with one exception, given linked procceses l1 and l2, when l1 experiences l2’s exit exactly as if l2 had called Process.exit/2 on l1 with its exit reason. Linked processes that exit by a :kill Remember that :kill is an untrappable exit when calling Process.exit/2? Its untrappability does not cascade to linked processes. {:ok, l1} = Linky.start() {:ok, l2} = Linky.start() Linky.link(l1, l2) Linky.trap_exits(l1) Process.exit(l2, :kill) true = Linky.alive_after_wait_for_death?(l1) Interestingly the reason received by the linked process is :killed not :kill, which would of course be trappable if sent via Process.exit/2. That is one explanation for :kills not cascasding through linked processes. {:ok, l1} = Linky.start() {:ok, l2} = Linky.start() Linky.link(l1, l2) Linky.trap_exits(l1) GenServer.stop(l2, :kill) true = Linky.alive_after_wait_for_death?(l1) It would be an explanation for kills not cascading, except it is perfectly possible, as above, for a process to exit with a reason:kill, and have the signal :kill trapped by a linked process. (Possible, but it would be an curious thing to actually do in production code.) {:ok, l1} = Linky.start() {:ok, l2} = Linky.start() Linky.link(l1, l2) Process.exit(l2, :kill) false = Linky.alive_after_wait_for_death?(l1) Of course if we are not trapping exits and a linked process is kiled, then the other processes also dies. Linked processes exiting with other reasons {:ok, l1} = Linky.start() {:ok, l2} = Linky.start() Linky.link(l1, l2) GenServer.stop(l2, :whatever) false = Linky.alive_after_wait_for_death?(l1) For completeness, the above code shows that if you are not trapping exits then linked processes will exit when the other exits, as long as the reason is not :normal. Linked (non GenServer / OTP) processes that simply exit {:ok, l1} = Linky.start() {:ok, l2} = Linky.start() spawny = spawn(fn -> receive do :bye -> IO.puts("Goodbye sweet world 😿") end end) Linky.link(l1, spawny) Linky.link(l2, spawny) Linky.trap_exits(l1) send(spawny, :bye) [true, true] = for pid <- [l1, l2], do: Linky.alive_after_wait_for_death?(pid) When any process’s function returns normally, then the process exits with the :normal reason. Linked processes behave accordingly: they receive a :normal exit notification from the Linked process if they are trapping exits; even if they are not trapping exits, they do not exit. This is why it’s safe, for instance, to start a task with Task.start_link/1 or spawn_link/1. A silly mistake I made that resulted in a process leak For reasons I created a helper process for a LiveView page in the mount/3 callback. I wrote something like def mount(_params, _session, socket) do {:ok, debouncer} = Debouncer.start_link(self(), 500) {:noreply, assign(socket, debouncer: debouncer)} end You can see the problem here, right? Yes, mount/3 is called twice: once when the initial html is rendered a and again when the socket connects, so an extra Debouncer is started. This would be a waste of cpu cycles, but no more, except that the initial rendering call is initiated by a Cowboy process that exits with a :normal reason. The Debouncer process does not exit, but stays orphaned and taking up resources. def mount(_params, _session, socket) do if connected?(socket) do {:ok, debouncer} = Debouncer.start_link(self(), 500) {:noreply, assign(socket, debouncer: debouncer)} else {:noreply, socket} end end The above mount/3 does not leak processes, as the LiveView’s socket process exits with {:shutdown, reason}, eg {:shutdown, :closed}, causing the linked Debouncer to also exit. It would be even safer, and proof against LiveView changes, to trap exits in Debouncer and voluntarily exit with a :stop return on the callback; I may do just that. I thought about adding this story at the beginning of the post, but I thought it might make it like one of those cooking articles that people complain about - the ones where you have to scroll though paragraphs of prose before getting to the actual recipe. (Also it doesn’t reflect well on me and people might get bored before reading this far.)Blogging with LiveBook2021-06-06T09:07:59+00:002021-06-06T09:07:59+00:00/elixir/2021/06/06/blogging-with-livebook<p>Last week I wrote <a href="/elixir/otp/2021/05/31/the-many-and-varied-ways-to-kill-an-otp-process.html">a post on killing OTP processes</a>, which was also a <a href="https://github.com/elixir-nx/livebook">LiveBook page</a>. The process for embedding LiveBook in a <a href="https://jekyllrb.com">Jekyll</a> blog was straightforward, if not entirely satisfactory. If you’re curious, or want to try it, here’s my current setup for this:</p>
<h2 id="directory-for-livebook-pages">Directory for LiveBook pages</h2>
<p>The standard Jekyll setup has a directory <code class="language-plaintext highlighter-rouge">_posts</code> for your blog posts. (The engine processes the contents of the directory to populate the <code class="language-plaintext highlighter-rouge">_site_</code> directory with your posts.). I created a <a href="https://github.com/paulanthonywilson/furlough/tree/c133d351989adf4ca7a73e17422d74d4a2318da3/_posts/livebook">livebook</a> directory under posts to house my LiveBook pages<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>.</p>
<h2 id="publishing-the-livebook-page-as-a-blog-post">Publishing the LiveBook page as a blog post.</h2>
<p>Once I’ve got my LiveBook page in <code class="language-plaintext highlighter-rouge">_posts_/livebook</code>, I still need to get it into the blog. A Jekyyll post needs some headers (not LiveBook friendly) and the content. To get the page embedded, I created new blog post and used the <a href="https://jekyllrb.com/docs/includes/"><code class="language-plaintext highlighter-rouge">include_relative</code> tag</a> to embed the page. The entire previous blog post (source <a href="https://github.com/paulanthonywilson/furlough/blob/c133d351989adf4ca7a73e17422d74d4a2318da3/_posts/2021-05-31-the-many-and-varied-ways-to-kill-an-otp-process.md">here</a>) is thus</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>---
layout: post
title: The many and varied ways to kill an OTP Process
date: 2021-05-31 11:02:22 +0100
author: Paul Wilson
categories: elixir otp
---
{% include_relative livebook/varied-ways-to-kill.livemd %}
</code></pre></div></div>
<h2 id="dealing-with-the-double-header">Dealing with the double header</h2>
<p>Jekyll adds the post title as a <code class="language-plaintext highlighter-rouge">h1</code> header. LiveBook also creates a <code class="language-plaintext highlighter-rouge">h1</code> header for the page title, so now there’s got two titles. I guess I could fix this by post-processing the LiveBook page, or doing fancy stuff with Jekyll but as no other posts should have <code class="language-plaintext highlighter-rouge">h1</code> in content I just hid it in the (s)css (<a href="https://github.com/paulanthonywilson/furlough/blob/c133d351989adf4ca7a73e17422d74d4a2318da3/_sass/minima/_layout.scss#L232-L238">source</a>).</p>
<div class="language-scss highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">.post-content</span> <span class="p">{</span>
<span class="nt">h1</span> <span class="p">{</span>
<span class="nl">display</span><span class="p">:</span> <span class="nb">none</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Admittedly, it’s a bit of a hack and probably mildly dubious from an accessibility point of view</p>
<h2 id="pointing-the-livebook-server-to-the-right-place">Pointing the Livebook server to the right place</h2>
<p>LiveBook servers can be kicked off from any directory and browser to pick up or save pages to any other directory. For convenience, though, I created a <a href="https://github.com/paulanthonywilson/furlough/blob/master/bin/live-blog">shell script</a> in this blog’s <code class="language-plaintext highlighter-rouge">bin</code> directory for launching a server pointing in roughly the right direction.</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/usr/bin/env sh</span>
<span class="nb">set</span> <span class="nt">-e</span>
<span class="nv">ROOT_PATH</span><span class="o">=</span><span class="sb">`</span><span class="nb">dirname</span> <span class="nv">$0</span><span class="sb">`</span>/../_posts/livebook
livebook server <span class="nt">--root-path</span><span class="o">=</span><span class="nv">$ROOT_PATH</span>
</code></pre></div></div>
<h2 id="satisfaction">Satisfaction</h2>
<p>The LiveBook embedded post looks ok, and fits with the rest of the posts in this blog. (Obviously it all could look a lot better than the basic theme, but that kind of thing is beyond my capabilities).</p>
<p>Of course, the blog post is not directly executable, though someone could download it from <a href="https://github.com/paulanthonywilson/furlough/blob/master/_posts/livebook/varied-ways-to-kill.livemd">here</a> and execute it locally.</p>
<p>It would be a lot better if the output of executing all the code in the page could be somehow captured and published, though I can think of all kinds of ways that this would not be straightforward. I may take a look sometime, though.</p>
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p><em>cough</em> I guess LiveBook <em>page</em> at the time of writing; there’s only one. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Paul WilsonLast week I wrote a post on killing OTP processes, which was also a LiveBook page. The process for embedding LiveBook in a Jekyll blog was straightforward, if not entirely satisfactory. If you’re curious, or want to try it, here’s my current setup for this: