yoshuawuyts

This post is part of a series: - Pull Streams (current) - Rust Streams Streams are an asynchronous abstraction that allows dealing with large data sets in small chunks, pushing bottlenecks into the IO layer. This usually leads to less memory cost and increased performance, which is a very good thing. In streams, data flows from a Source, through a bunch of Through streams, into a Sink: txt txt ┌────────┐ ┌─────────┐ ┌────────┐ │ Source │──▶│ Through │──▶│ Sink │ └────────┘ └─────────┘ └────────┘ Copy Node has shipped streams as part of its standard library since its early days. However with each release, new features, APIs and concepts were added, making the current implementation very unwieldy. In my years as a Node developer I've only met a handful of developers that felt comfortable using the current version of Node streams. In an ideal world streams would be used as much in Node as pipes are used in shell. ## Enter pull-streams pull-stream is an alternative model for streams created by Dominic Tarr. Like Node streams, it has a concept of backpressure. This means that instead of a source pushing out data as fast as it can, the consumer stream pulls data once it's ready to handle more. This leads to a program never holding more data in memory than it needs. The Node streams source is well over 1200 lines, without even accounting for dependencies. The pull-stream source is just 28 lines, which is a whopping 0.4kb minified: js js module.exports = function pull (a) { if (typeof a === 'function' && a.length === 1) { return function (read) { var args = [].slice.call(arguments) return pull.apply(null, args) } } var read = a var n = arguments.length var i = 1 if (read && typeof read.source === 'function') { read = read.source } for (; i < n; i++) { var s = arguments[i] if (typeof s === 'function') { read = s(read) } else if (s && typeof s === 'object') { s.sink(read) read = s.source } } return read } Copy Don’t be fooled by the simple exterior though, pull-stream provides the same functionality as Node streams do. With fewer lines of code there’ll be less bugs, and room to optimize every last bit of code. ## Pull-stream types In pull-streams there are 3 types of streams: source, through and sink. In order to let data flow, a source and sink must be connected. Through streams are combinations of sources and sinks, making every connection in the pipeline a source and a sink that talk to each other. Conceptually it looks like this: txt txt ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐ │Source│──▶│ Sink │ ┌▶│ Sink │ ┌▶│ Sink │ └──────┘ ├──────┤ │ ├──────┤ │ └──────┘ │Source│─┘ │Source│─┘ └──────┘ └──────┘ Through Through Copy If a source and a through stream are connected they will not start emitting data until a sink is attached at the end. Likewise, if a through and sink are connected, they will not start flowing data until a source is attached at the start. This allows composition of arbitrary streams into pipelines similar to what pumpify provides for Node streams. ## Composition pull-streams use the pull() function to combine sinks and sources. Because sinks connect to sources, any number of streams can be connected. It's functional composition all the way. Duplex (through) streams are objects that have a .source and .sink properties on them. The following three methods of connecting pull-streams are equivalent: js js pull(a.source, b.sink) pull(b.source, a.sink) Copy js js b.sink(a.source) a.sink(b.source) Copy js js pull(a, b, a) Copy ## Helper functions pull-stream ships with helper functions such as asyncMap that make common interactions trivial. See the docs for available sources, sinks and throughs. Let's create a basic map-reduce pipeline using asyncMap where we asynchronously fs.stat an array of files, and gather the results in an array: js js const pull = require('pull') const fs = require('fs') const source = pull.values([ './file1', './file2', './file3' ]) const through = pull.asyncMap(fs.stat) const sink = pull.collect(function (err, array) { if (err) return console.error(err) console.log(array) }) pull(source, through, sink) Copy Because under the hood we're just composing functions, the overhead of doing this is reduced to a bare minimum. ## Error handling In Node streams errors don't propagate through .pipe() chains. It's therefore common practice to either use helper libraries or attach a .on('error') listener to every stream. Getting errors wrong is not a great feeling. In pull-streams errors are passed into the callback, which grinds the whole stream pipeline to a halt. It's again the familiar API of cb(err) for an error and cb(null, value) for success. Here's a source stream that returns a single fs.stat value: js js const fs = require('fs') function readFile (filename) { var read = false // keep track if we've executed the action return function source (end, cb) { if (end) return cb(end) // stop signal was passed, stop doing things if (read) return cb(true) // we're done with fs.stat, stop doing things fs.stat(filename, function (err, file) { if (err) return cb(err) // ohey, an error read = true // mark that we've executed the action return cb(null, file) // pass value down the pipeline }) } } Copy ## Wrapping it up And that's it. I could keep whipping out pull-stream examples, but I think we've made our point: streams are a cool idea; pull-stream is a neat implementation. If you're keen to learn more, take a look at the links below. I hope this was useful; give pull-stream a try let me know how you go! -Yosh - twitter/yoshuawuyts - github/yoshuawuyts ## See Also - pull-stream - pull-stream-examples - pull-stream/spec - pull-stream/sources - pull-stream/sinks - pull-stream/throughs

Hey everyone! Welcome to my new blog. Stretch your legs. Make yourself at home. ## Why? I used to be quite active on Medium.com, but I don't think the site has been changing for the better, and want to try something new. Something that's my own. Something where I have the creative freedom to structure it however I like. So this is it. ## Is this a dat:// site? It's not. At least: not yet. Perhaps we'll get there eventually, but for now I'm happy to build a static site, and distribute it from my own server. I've always wanted to be better at managing servers, and this is a fun addition to my growing collection of things running on a single 5/euro month instance. Keeping things static means performance is reliable, and reliability is nice for hobby projects. A long story short: perhaps I'll make the jump to dat:// eventually. But not right now. ## Source You can find the source on github.com/yoshuawuyts/blog, and the server configuration on github.com/yoshuawuyts/server. The theme is Zola After Dark, but I expect it won't be long before I start tinkering with it. ## Wrapping Up Thanks for reading! I hope you visit again soon!

Last year was pretty good. I worked on choo, datrs, and got 2 cats. I started working out (more) regularly, and towards the end of the year even consistently stayed on top of my email. Also I've cooked a ton of food, and definitely feel like I'm a better cook than I was at the start of the year. In this post I want to share my plans & goals for this year. ## No more injuries I've had problems with my wrists since I was about 16 years old. This has mostly been due to bad posture while using laptops, and working too long and too hard. Last year it never got so bad I couldn't work, but I'm definitely not healed yet. This year I'd like to continue working towards making a full recovery. This means probably working less long hours, taking more breaks, and not working on weekends. Also paying more attention to my workrave timers. Being better about timing work is a good idea by itself. But having a goal motivating is the little extra nudge that really makes me want to commit to this. ## p2p datrs has been really fun to work on last year. It was my first full-time Rust job, and I'm really pleased with the results. This year I'd like to get the networking stack going, based on async/await and romio. There's is useful for many reasons, but in particular I'm excited for the possibility of helping the peermaps advance. Emphasis on "possibility". Regardless going to be useful to provide feedback on async/await by building out more networking implementations outside exclusively HTTP contexts. I hope to have time to work this in Q2/Q3 this year, on the side. However if you'd like to get involved to help it advance more actively I'd be more than happy to take a more managerial role and help people move in the right direction! ## Web Services Because Rust's async/await story isn't stabilized yet, the ecosystem on top of it still needs a lot of work. This year I'll be working on building out modules that work on on top of async Rust. On the one hand there's a ton of protocol work that's in various stages of completion. HTTP/2, SSE, HTTP/3 crates exist, but they don't feel quite as polished as stdlib's TCP interface. We should probably have support for these, which we can likely get in the near term by wrapping existing C libraries. There's also the higher-level web-app stories. It don't feel like we have a good story yet for generating REST docs, reusable middleware, and authentication. I feel there's a lot of room to experiment and improve this space! I think it'd be great if we could tackle these problems with the rust-net-web WG. But this requires us to become better at organizing. Which is why I think this is a good opportunity to become more involved, and help guide this along! I've already started working on this, and plan to continue on it for the foreseeable future! ## Frontend I'm very curious if Rust can help us build better interfaces on the Web. To me it's clear that it has potential, but it's not clear yet what the abstractions should look like. I'd love to contribute this year to the Rust WASM by leveraging my knowledge of building choo, and helping the ecosystem succeed. I've already started joining the WASM WG meetings, and plan to become more involved as the year goes on. ## Relationship My partner is more concensious than I am, and that's sometimes hard in our relationship. Especially when it comes to doing chores around the house, I'm usually less organized about most things. (Except the kitchen. I run a tight ship.) I want to be better at managing the cat, the house, and the other cat this year. Also be better about planning trips and weekend plans. I feel we're not exceptionally out of balance, but it's clear that there's room to do better. So I should take this as a chance to learn and improve. ## Content I want to become better at producing content also. Write more often, and become increasingly clear. When I do video streams, have a plan and curate the output better. Make it feel like the things I make are recognizable. I haven't cared about this much in the past; but I think paying a bit more attention to production quality is another way of improving at communication. And that seems like something worth investing in. ## Wrapping Up And that's in broad strokes what I was thinking of working on this year. I don't think they're spectacular plans. But I feel they're achievable. And I'm pretty excited to make them happen this year!

Hey everyone! Two days ago I released my plans for 2019. Earlier today I wrote about what I'd like to see for Rust WASM in 2019. This is my post about what I'd like to see happen with Rust in 2019. However unlike my two previous posts, I'd like to keep this one short and sweet. ## Speaking is silver I've been drafting this post for the past two months. However none of it felt right. I think it's probably because others have put my feelings into words better than I'm currently capable of. There are two posts I'd like you to read: First off is Boats' post on organizational debt. It touches on how we organize ourselves, and the problems we're facing. I feel it distills so well which challenges we face as an organization, and feel it ought to set the tone on what to prioritize in this year's roadmap. The second post is Gattozzi's post. It feels very personal and it's focused around his experiences with the people in Rust. I recognize myself a lot in those experiences. The community is changing, and we'll have to think of how we plan to change with it. --- Have you read the two posts above? Yes? Great! Thank you for your time. From here on out it's desserts only. ### No silos, no heroes My main joy in programming stems from curiosity. I love learning new things, figuring out how things work, and trying to improve them. The more open systems are, the better this works. Which is why I got involved with open source in the first place, and Rust with its RFC process later on. However this becomes harder when systems are closed. When decision making is opaque, code is hard to understand, or people use their status to justify their actions. Every community has to deal with this, and Rust is no different. I encourage everyone to reflect on who they look up to, and why. Which pieces of code we think are cool, but don't understand. Which crucial projects are lead by people, not teams. We should think about what kind of community we want to be, and how we can ensure we can get there, together. ### Inclusive behavior I often think about this tweet by Aurynn: > Diversity is an emergent effect of inclusive behaviour. I'd describe "inclusive behaviour" as behaviour that makes people feel valued, welcome, and not threatened. Perhaps even simpler: behaviour that doesn't actively exclude people (except people that exclude others). I know we'd all like to believe the Rust community is a shining example of what a programming community should look like. But have we truly reached the pinnacle of behavior. Is there collectively no more room for improvement? This definitely ties back into the two posts I linked to at the start of this post. But instead of proposing directions for the core team to take: I'd like this to be a call for each of us to reflect on how we interact with other people. Being welcoming is our collective responsibility. The way we behave reflects on our peers. And the way our peers behave is a reflection of us. ## Wrapping Up And that's it for this year! I hope this has been useful! I'm starting to get a taste for writing posts, so perhaps I'll write about more things going forward. But anyway, it's late here, so I'm going to call it a night for now. Hope you're well / have a good eve!

Two days ago I wrote about what I plan to work on this year. This is my post about what I'd like to see happen with WebAssembly and Rust in 2019. ## Status quo 2018 has been a really cool year for WASM and Rust. Back in February the WASM toolchain became usable. Around the same time both wasm-bindgen and wasm-pack were kicked off (I remember the dates because of this gem). I think it's interesting to point this out, because it means that at the time of writing Rust's defacto WASM toolchain is about 9 months old. That's not a long time for a project that is now considered standard. It's super impressive how much the WASM WG has accomplished last year! Let's take a look at what might be possible to accomplish this year. ## fn main The narrative of the WASM WG's tooling in 2018 has mostly revolved around allowing people building browser applications to gradually introduce Rust into their workflows. This has been great for people looking to improve parts of their existing JavaScript codebase. I'd like Rust's WASM story to focus on building an integrated experience this year. Even though WASM is not intended to replace JavaScript, I think we can learn a lot by pushing WASM as far as we can by itself. I think this should give us insights in what is missing, and what needs improving, leading us to create a better Rust WASM experience for every workflow. I think a good goal is to aim to add first-class support for standalone WASM binaries. Standalone meaning: the Rust code drives the application logic, rather than having JavaScript call into Rust library functions. For example [wasm_bindgen(start)] can be used as an application entry point, and the resulting binaries can be served with a static web server. But neither of them feel as nice as Rust's own fn main and cargo run. ## Build an ecosystem Speaking of ecosystems, we should probably build one! There's a few points I think are important: 1. Don't create branded silos. Branding might perhaps be useful to achieve fame. But if we truly want Rust's WASM story to succeed we should think of ways to collaborate instead of carving out territory. 2. Centralize discovery. We're trying to build an ecosystem from scratch. Having a place where module authors can share their results will likely allow us to coordinate without much overhead. Something that for example might work is for the WASM WG to create an awesome list under the GitHub org, and liberally accepting contributions. 3. Encourage experimentation. I think it's way too early in Rust's lifecycle to know what the right way is to do things. Even more so for Rust WASM. I think we should encourage experimentation and sharing results early. The lower we can make the boundary to experiment, the likelier it is that people will get involved, and in turn the likelier the ecosystem will grow in both size and quality. ## Keep polishing This is the "miscellaneous" part of my post. There's some things I've noticed that are missing, or I wish we had in no particular order: - wasm-bindgen doesn't generate source maps yet. Having them would be excellent for debugging. - The story around async and WASM seems only partly executed so far. Rust Futures can be converted to JS Promises and hooked into the JS event loop, but running your own multi-threaded loop seems to involve a lot of manual work still. - Rust WASM artifacts are single binaries which are hard to cache and incrementally load. There's probably exploration possible around code splitting and dynamically loading. - Setting up a WASM project requires quite some boilerplate. It'd be nice if we could find ways to reduce this. - Passing errors to and from JS isn't as smooth as it could be. ## Wrapping up And that's more or less what I'd like to see from Rust WASM this year. I feel Rust's WASM story is very much on the right patch already, and I hope this will only keep improving in 2019. I hope this has been useful. Thanks for reading!