Chapter Feedback from Student Perspective

[sgiannuzzi39]

Hi all -- a couple other undergraduates and I have been working through the book and have found it has been super helpful! That being said, there are a few places where we got confused or would find further details helpful. I figured I would start this discussion so students could similarly contribute their feedback, and perhaps experts could answer the questions we have!

[sgiannuzzi39]

Chapter One - Introduction

(Notes are included on the PDF linked below, but here is a brief overview of our comments)

• We had a hard time pulling out key terms and definitions. Bolding certain relevant terms and/or creating a dictionary at the beginning of the chapter would be super helpful -- for instance, in this chapter, we could include "ubiquitous computing" as a key term. For more interactivity, we could have the definition pop up when you hover over bolded words.

MLS Textbook - Chp 1.pdf

[profvjreddi]

Avoided bolding, but tweaked confusing sentences.

[sgiannuzzi39]

Chapter Two - ML Systems

(Notes are included on the PDF linked below, but here is a brief overview of our comments)

• When looking at the learning objectives, we feel like we did not take away the second bullet point. A lot of the key terms included in this point were not explicitly stated so we didn't know what they were referring to.
• Figure 2.1 seems to show that TinyML is a type of EdgeML. Is that the case? If so, can you specify that in writing?
• In 2.2, we wish that Cloud ML "Definition" came before "Characteristics." More specifically, we get confused when we define Cloud ML as a blend of powerful blend of high-capacity servers versus the ML actual being defined. Maybe this would be helped by building out how Cloud ML excels in data processing and model training.
• By the time 2.4 comes around, we found ourselves confused about how Tiny differed from Edge. In the comparison section, it said that Edge operates on a local network -- is that the difference? If so, can this be specified a bit more?
• IoT is mentioned a lot in this section but not fully explained.
• After reading 2.4.1, we were still confused on what a microcontroller does.

MLS Textbook - Chp 2.pdf

[profvjreddi]

When looking at the learning objectives, we feel like we did not take away the second bullet point. A lot of the key terms included in this point were not explicitly stated so we didn't know what they were referring to.

Good catch - fixed it!

Figure 2.1 seems to show that TinyML is a type of EdgeML. Is that the case? If so, can you specify that in writing?

Done.

In 2.2, we wish that Cloud ML "Definition" came before "Characteristics." More specifically, we get confused when we define Cloud ML as a blend of powerful blend of high-capacity servers versus the ML actual being defined. Maybe this would be helped by building out how Cloud ML excels in data processing and model training.

Done, added a bunch of new text.

By the time 2.4 comes around, we found ourselves confused about how Tiny differed from Edge. In the comparison section, it said that Edge operates on a local network -- is that the difference? If so, can this be specified a bit more?

Clarified this further by adding a figure from prof. @songhan's papers.

IoT is mentioned a lot in this section but not fully explained.

Done!

After reading 2.4.1, we were still confused on what a microcontroller does.

Added a hyperlink for additional information.

[sgiannuzzi39]

Chapter Three - DL Primer

• In 3.2 Neural Networks, we feel like we would've benefitted from more detail about weights, biases, and activation functions used by a Perceptron to create an output.
• In 3.2.2 Multilayer Perceptron section, we didn't fully understand the concept of layers until we watched the included videos. If students are intended to understand these ideas without help from video, perhaps go into more detail?
• After reading the Backpropagation section, we still found ourselves confused about how this algorithm benefits deep neural networks
• We wish there was more detail in the CNN section. Specifically, how are CNNs different from multilayer perceptrons?
• In 3.2.5 Choosing Traditional ML vs DL, we were under the assumption that deep learning is a kind of ML. Could you clarify this in the text?

Machine Learning Systems - 3 DL Primer.pdf

[sgiannuzzi39]

Chapter Four - AI Workflow

• In 4.3 Roles & Responsibilities, it would be helpful if it included which part of the lifecycle each role worked on.
• No conclusion?

Machine Learning Systems - 4 AI Workflow.pdf

[sgiannuzzi39]

Chapter Five - Data Engineering

• Best chapter yet in our opinion! We would love to see other chapters go similarly in depth for their topics
• Perhaps include more graphics to break up the text -- this is a very text-heavy chapter

Machine Learning System - 5 Data Engineering .pdf

[sgiannuzzi39]

Chapter Six - AI Frameworks

• The first thing we wanted to mention here (but applies to whole textbook!) is that, while this is an interactive textbook, a lot of students are still going to download PDFs and export the file to Goodnotes/Notability for note-taking. It would be good to ensure that the textbook supports this functionality -- we noticed that some of the text seems to disappear/become too light when you download the PDF.
• Another thing we noticed (but again applies to the whole textbook) is that there is a sense of incoherency from section to section or chapter to chapter. The tone and style shifts noticeably between authors, and the chapters don't seem to build on each other. For instance, the concept of layers (6.4.2 Graph Definition) has already been defined in Chapter 3, but feels redefined here. The same goes for synthetically-expanded datasets (6.4.4 Data Augmentation) and loss functions (6.4.5 Optimization Algorithms).
• There is a lot of text in this section. Our first suggestion is to pare it down a bit; things began feeling a bit repetitive (especially between 6.8 Examples and 6.9 Choosing the Right Framework), and some things just felt like they could've been excluded from the chapter (like 6.7 Embedded AI Frameworks). Adding in images and graphs could also help space out the text a bit.
• After reading 6.4.1 Tensor Data Structures, we felt like we understood what a tensor is, but not how it is used. We would love more detail on that.

Machine Learning Systems - 6 AI Frameworks.pdf

[profvjreddi]

Thanks for the candid feedback! This is very helpful. Will certainly work on it!

…

On Wed, Jun 12, 2024 at 12:26 PM, sgiannuzzi39 ***@***.***> wrote: *Chapter Six - AI Frameworks* - The first thing we wanted to mention here (but applies to whole textbook!) is that, while this is an interactive textbook, a lot of students are still going to download PDFs and export the file to Goodnotes/Notability for note-taking. It would be good to ensure that the textbook supports this functionality -- we noticed that some of the text seems to disappear/become too light when you download the PDF. - Another thing we noticed (but again applies to the whole textbook) is that there is a sense of incoherency from section to section or chapter to chapter. The tone and style shifts noticeably between authors, and the chapters don't seem to build on each other. For instance, the concept of layers (*6.4.2 Graph Definition*) has already been defined in Chapter 3, but feels redefined here. The same goes for synthetically-expanded datasets (*6.4.4 Data Augmentation*) and loss functions (*6.4.5 Optimization Algorithms*). - There is a lot of text in this section. Our first suggestion is to pare it down a bit; things began feeling a bit repetitive (especially between *6.8 Examples* and *6.9 Choosing the Right Framework*), and some things just felt like they could've been excluded from the chapter (like *6.7 Embedded AI Frameworks*). Adding in images and graphs could also help space out the text a bit. - After reading *6.4.1 Tensor Data Structures,* we felt like we understood what a tensor is, but not how it is used. We would love more detail on that. Machine Learning Systems - 6 AI Frameworks.pdf <https://github.com/user-attachments/files/15809340/Machine.Learning.Systems.-.6.AI.Frameworks.pdf> — Reply to this email directly, view it on GitHub <#256 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABT6DFFSEGNAFRRHSJNIOLTZHCHELAVCNFSM6AAAAABJCT4RKWVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TONJVGMZDK> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** .com>

[sgiannuzzi39]

Chapter Seven - AI Training

• We found that Chapter Three and Chapter Seven cover almost the exact same topics, though Chapter 7 went into more depth. We recommend either consolidating the two or just letting Chapter Seven stand on its own.
• This chapter is the longest we've encountered so far---towards the end, we were all having a hard time retaining the information. We felt like we held on through Learning Objectives 1-4 , but beyond that our comprehension had dwindled. I believe this has more to do with the length of the text rather than the actual writing, so to fix this we recommend either cutting down the writing or splitting it into two chapters.
• There are a lot of equations in this chapter, but (at least for the first couple of equations) no context for variables or symbols that might be foreign to students. Additionally, the equations didn't appear when exporting the PDF, which we might want to change for students who prefer to take notes.
• Also for students who download PDFs, would there be a way to hyperlink the videos included so that they can be accessed from PDF format?
• We like the bolding of key terms in this chapter! Would be helpful to see this in other sections of the book.
• In the introduction, explaining some real-world implications/examples would help ground readers a bit more.
• 7.4 Training Data, 7.6 Hyperparameter Tuning, and 7.9 Activation Functions were all really long sections. A good rule of thumb is to have every section be ~2 pages, just so readers can feel like they're progressing through the chapter a bit more (good for student morale!). We found ourselves getting a bit bogged down beyond that.
• In 7.4 Training Data, there is a "Data Leakage Between Sets" section and "Information Leakage Between Sets" section. Are these two things different? We would love more clarification on that.
• To help minimize text, it seemed to us like 7.4.2 Common Pitfalls and Mistakes and 7.6.3 Auto Tuners could have been turned into tables. The tables in this book have really been a highlight for us, and are a great way to pull information easily.

Machine Learning Systems - 7 AI Training.pdf

[sgiannuzzi39]

Chapter Eight - Efficient AI

• In terms of learning objectives for this chapter, we felt like we generally got most of them! However, we wanted to point out that the first learning objective seemed very similar to the fourth. Also, the last two learning objectives seemed to be less captured--we weren't sure what "nuances" were being referred to in the 6th bullet point, and don't remember any mention of "environment" (as mentioned in the 7th bullet point).
• The formatting seemed a bit inconsistent--for instance, 8.2 The Need for Efficient AI seemed like it would benefit from the formatting maintained in the next section (8.3 Efficient Model Architectures) where terms like "TinyML" or "Edge AI" are bolded with colons prior to the paragraphs explaining them. Overall, we think the chapter would benefit from clearer subtitles and subsections with more bolded/header differentiations.
• The concept of "Pruning" in 8.4 Efficient Model Compression has been mentioned a couple times up until this point in the book (and we go on to redefine it in Chapter 9). The constant redefinitions make the book feel fragmented--perhaps when we mention these kind of terms that are mentioned/defined in multiple places, we should include hyperlinks to other sections where it is defined more fully, or a kind of "See Chapter 9 for more information" caption to make reading more cohesive. Another example of this is "FLOPs" (defined in 8.7 Evaluating Models), which was just mentioned in Chapter 7 but not defined; perhaps we should refer Chapter 7 readers to this section in case they are confused.
• Again, bullet points in the table in section 8.6.1 Numerical Formats would help readers more easily differentiate lines of text.

Machine Learning Systems - 8 Efficient AI.pdf

[sgiannuzzi39]

Chapter Nine - Model Optimizations

• We feel like we got all of the learning objectives! Maybe bring the 5th bullet point up one (so it's the fourth) to stay in line with the chapter layout.
• Again, pruning is defined again in 9.2.1 Pruning. Like we said with Chapter 8, I would make sure these definitions are cohesive and not as duplicated throughout the chapters.
• We found ourselves getting confused on the formatting in section 8.2 Efficient Model Representation. For instance, maybe bold subtitles like "Overview" or "Structured Pruning", or make their font bigger so they are differentiated from their subsections (like "Components"). I think especially with these longer sections, it's important to differentiate subsections for readers
• In 8.2 Efficient Model Representation, "sparse weight matrices" were mentioned several times, the first of which was in Table 9.1. We never felt we got a concrete definition on this term, and would love some more explanation.
• A lot of 9.3.1 The Basics already feels defined in Chapter 8 - in fact, the table in this section is almost the exact same as the one in Chapter 8.
• We wish 9.3.1 The Basics went into more detail surrounding "mantissa, exponent, and sign" categorizations (as shown in Figure 9.13).
• A couple of silly typos (all shown in-line in PDF): under Table 9.1, it says "we have exapmles" instead of "we have examples." Under the "Lottery Ticket Hypothesis" section, the second paragraph has the sentence, "They empirically discovered by Jonathan Frankle and Michael Carbin." Section 9.4.3 Kernel Optimizations says "machine learning models onf resource-constrained devices" instead of (we assume) "on." Finally, in 9.5.3 Hardware Optimization Libraries, a subtitle reads, "Sparsity (ADD SOME LINKS INTO HERE)" - we imagine that was intended for a later edit.
• Visualizations were great in this chapter!
  Machine Learning Systems - 9 Model Optimizations .pdf

[sgiannuzzi39]

Chapter 10 - AI Acceleration

• In terms of learning objectives for this chapter, we did not feel like we walked away with the fourth bullet point or the sixth bullet point.
• This was an interesting chapter, but the tone felt like an academic paper. We feel like there is a more accessible way to phrase a lot of the information, especially for students who have not encountered a lot of the vocabulary before. Perhaps also emphasize why this is important for students to be learning.
• There was a lot of text in this chapter, which made it hard for us to fully grasp everything. Maybe find some way to cut down on length--there were a few places in the chapter where bullet points would have sufficed (like in the Disadvantages/Advantages section of 10.3.2 Field-Programmable Gate Arrays (FPGAs), 10.3.3 Digital Signal Processors (DSPs), 10.3.4 GPUs, 10.3.5 CPUs, etc.). Also, including tables in the place of text is often a better way to get the point across, rather than doing both table and text.
• 10.6 Benchmarking Hardware feels unnecessary, given that the whole next chapter is about benchmarking. Perhaps include one or the other, but both do not seem necessary.
• We didn't understand what the table in 10.3.1 Application-Specific Integrated Circuits (ASICs) was trying to get across, especially the "Notes" column. Some more clarification on this would be great! (Also the table does not have a number or way to reference it)
  Machine Learning Systems - 10 AI Acceleration.pdf

[sgiannuzzi39]

Chapter 11 - Benchmarking AI

• We felt like we walked away with all learning objectives except for the seventh.
• This chapter was interesting because it tried to broach a large topic (benchmarking) by tackling specific cases. While this was helpful, it resulted in a lot of the text being repetitive, as the same ideas were being re-introduced with only slightly different contexts. Also, benchmarks have been mentioned/defined several times up until this point, normally to help explain benchmarks for the topic of each chapter. Because of these two things, we feel like this chapter might be unnecessary. Rather, you can build up the explanation of benchmarks in each chapter, giving relevant information on benchmarks while still sticking to the topic of the chapter. Readers would still walk away with understanding of benchmarks in specific capacities, altogether giving them a general idea of what benchmarks are.
• Some repetition in explanation that we noticed were across 11.4.4 Training Benchmarks, Metrics section and 11.4.5 Inference Benchmarks, Metrics section. These ideas were generally the same, and explanations of the each metric in both places did not feel necessary. Also, in 11.4.4 Training Benchmarks, MLPerf Training Benchmark is explained twice (and each explanation has a paragraph dedicated to it).
• This chapter was also long; a potential place to cut down on text could be 11.5.1 Historical Context. The background information is nice, but not strictly necessary.
• We didn't understand Figure 11.5 (sorry!)
• The 11.5.3 Lessons Learned section felt like more of a conclusion---perhaps it should go towards the end of the chapter, rather than the middle.
• Figure 11.7 is right under the 11.7 The Trifecta heading, but does not demonstrate the trifecta. Perhaps switch out this image with something more relevant?
  Machine Learning Systems - 11 Benchmarking AI.pdf

[sgiannuzzi39]

Chapter 12 - On-Device Learning

• One of the things we thought of while reading this chapter was that it would be helpful to have some note-taking functionality on the website. If readers could highlight on the site or make comments, we wouldn't need to download PDFs to take notes externally. This would solve a lot of formatting problems we encounter once transferring to PDF format.
• This chapter echoes a lot of the information from Chapter 2. Perhaps cut down Chapter 2 and put this chapter afterwards? I have highlighted within the PDF attachment below specific areas that feel repetitive.
• Figure 12.2 has already been used in the textbook.
• The bullet point format used in this chapter was fantastic!
• Video 12.2 had a playback error when we clicked on it.
• We didn't really understand Figure 12.6--I would either cut or replace this image.
• We wished there was a bit more explanation on IID in 12.5.5 Handling non-IID Data.
  Machine Learning Systems - 12 On-Device Learning.pdf

[sgiannuzzi39]

Chapter 13 - ML Operations

• One of the things we realized when reading this chapter was that a "Revision" section at the beginning of each chapter could be helpful. The section would consist of brief explanations of topics that have been mentioned before in the textbook, but the reader might have forgotten in the meantime. A lot of these topics build off of each other and we are constantly seeing a re-introduction of things we covered before--this "Revision" section inclusion could help cut down on that duplication of explanation.
• We felt like we got everything but the last Learning Objective bullet point.
• The first paragraph of this section offers some roadmapping as to where the chapter is going to go--this kind of roadmapping has been included in some chapters but not others. We would recommend standardizing this across all chapters, or not including it in any.
• Roadmapping is also included in the beginning of 13.3 Key Components of MLOps; the double roadmapping from the introduction to here is not necessary.
• In 13.2 Historical Context, specifically the portion explaining DevOps, felt a little unnecessary and seemed like it had the potential to confuse readers. If looking to shorten, I think this section could be cut.
• Figure 13.3 has been included in previous chapters, and is included twice in this chapter (also seen in Figure 13.4)
• The 13.5 Roles and Responsibilities section feels very similar to Chapter 4's 4.3 Roles and Responsibilities section--I don't think we need both.
• I also feel like we've covered 13.6 Embedded System Challenges in earlier chapters; it feels like we could cut this.
• (I know I'm being repetitive at this point but) 13.7 Traditional MLOps vs. Embedded MLOps also feels like it's been implicitly covered in earlier chapters, and you could cut this section.
• 13.9 Case Studies felt a little more in-depth than was necessary. As this is a long chapter, perhaps cut some of this section?
  Machine Learning Systems - 13 ML Operations.pdf

[profvjreddi]

Thanks for the feedback, keep it coming! I've added @jason to this thread to help us address all your feedback @sofia Vijay Janapa Reddi, Ph. D. | John L. Loeb Associate Professor of Engineering and Applied Sciences | John A. Paulson School of Engineering and Applied Sciences | Science and Engineering Complex (SEC) | 150 Western Ave, Room #5.305 | Boston, MA 02134 | Harvard University | My Website <http://scholar.harvard.edu/vijay-janapa-reddi> | Google Scholar <https://scholar.google.com/citations?hl=en&user=gy4UVGcAAAAJ&view_op=list_works&sortby=pubdate> | Edge Computing Lab <https://edge.seas.harvard.edu> | Book Meeting <https://fantastical.app/vjreddi/> | Contact Admin <https://scholar.harvard.edu/vijay-janapa-reddi/contact> |

…

On Fri, Jul 05, 2024 at 2:01 PM, sgiannuzzi39 ***@***.***> wrote: *Chapter 13 - ML Operations* - One of the things we realized when reading this chapter was that a "Revision" section at the beginning of each chapter could be helpful. The section would consist of brief explanations of topics that have been mentioned before in the textbook, but the reader might have forgotten in the meantime. A lot of these topics build off of each other and we are constantly seeing a re-introduction of things we covered before--this "Revision" section inclusion could help cut down on that duplication of explanation. - We felt like we got everything but the last Learning Objective bullet point. - The first paragraph of this section offers some roadmapping as to where the chapter is going to go--this kind of roadmapping has been included in some chapters but not others. We would recommend standardizing this across all chapters, or not including it in any. - Roadmapping is also included in the beginning of *13.3 Key Components of MLOps*; the double roadmapping from the introduction to here is not necessary. - In *13.2 Historical Context*, specifically the portion explaining DevOps, felt a little unnecessary and seemed like it had the potential to confuse readers. If looking to shorten, I think this section could be cut. - *Figure 13.3* has been included in previous chapters, and is included twice in this chapter (also seen in *Figure 13.4*) - The *13.5 Roles and Responsibilities* section feels very similar to Chapter 4's *4.3 Roles and Responsibilities* section--I don't think we need both. - I also feel like we've covered *13.6 Embedded System Challenges* in earlier chapters; it feels like we could cut this. - (I know I'm being repetitive at this point but) *13.7 Traditional MLOps vs. Embedded MLOps* also feels like it's been implicitly covered in earlier chapters, and you could cut this section. - *13.9 Case Studies* felt a little more in-depth than was necessary. As this is a long chapter, perhaps cut some of this section? Machine Learning Systems - 13 ML Operations.pdf <https://github.com/user-attachments/files/16112583/Machine.Learning.Systems.-.13.ML.Operations.pdf> — Reply to this email directly, view it on GitHub <#256 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABT6DFFVXFVQVIWDGV2KXD3ZK3NPNAVCNFSM6AAAAABJCT4RKWVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TSNZQG4ZDQ> . You are receiving this because you commented.Message ID: ***@***.*** com>

[jasonjabbour]

Thanks for the detailed feedback @sgiannuzzi39!

@profvjreddi I'll address the feedback in order starting from Chapter 3. Opened an issue to keep track of tasks here #315

[sgiannuzzi39]

@jasonjabbour thank you! I will keep adding updates to this Discussion board unless you prefer otherwise

[profvjreddi]

Yes, please continue to do that here so that we can track all the updates. And as we start working on them @jasonjabbour will create a new branch that he and I can work on together.

[sgiannuzzi39]

Chapter 14 - Security & Privacy

• We felt like we got all Learning Objectives except for the last one (we don't really recall a discussion of electrical, firmware, etc.)
• One thing that would super helpful across chapters would be some common formatting standard. This goes for both layout and more nitty-gritty formatting things like bolding subheaders, font size, indentations, etc. For instance, a lot of chapters have Historical Precedence sections (like this one), but some include them in the beginning and other in the middle/end. Some chapters include a "Terminology" section at the beginning (which is super super super helpful!) while others do not.
• This chapter again begins with a roadmap of where the chapter is going -- this should be included in all chapters or none of them (again, a standardization thing)
• There are a lot of terms in this chapter that are redefined from previous chapters or just stated with no real context. This makes us a think that an Index to the book could be really helpful -- we could include terms like GDPR (which is again redefined in this chapter) that we can allow users to reference as needed. As is, too many terms are redefined or not given any background, which really detracts from comprehension.
• There are nine case studies in this chapter. We think that some should be cut, as reading all nine gets a little tedious. Importantly, the formatting/naming of the case studies should be standardized. Some are just titled "Case Study" (like in 14.4.1) while others are numbered like "Case Study 1" (14.4.2) and others are given names like "Case Study: Apple's Face ID" (14.6.2) or "Case Study - Performance-Based Data Minimization" (14.7.4). We think that the naming with a colon is the best way to differentiate the studies (i.e. "Case Study: Apple's Face ID"), but really any standard practice works.
• We think that one of the three historical precedent examples in 14.3 Historical Precedent could be cut to minimize unnecessary reiteration; the third (14.3.3 Mirai Botnet) seems the least necessary.
• Generative Adversarial Networks (GANs) in section 14.4.3 Adversarial Attacks has been defined and explained in previous chapters -- again, this would be a good inclusion in some kind of Index, so we don't have to keep redefining terms that have already been explained. Later in this same chapter, we see it defined again in basically the exact same manner in section 14.8.6. Synthetic Data Generation -- we should probably cut this duplication.
• Figure 14.3, 14.4, and 14.5 were hard to understand. We recommend replacing them.
• 14.6 Embedded ML Hardware Security introduced a lot of acronyms and terms. We didn't think all of them were necessary and were likely to confused readers.
• 14.6.3 Hardware Security Modules "Benefits" section could be reformatted into a table to improve readability.
• 14.7.2 Applicable Regulations redefined a bunch of regulations that have been defined earlier in the textbook (like GDPR and HIPAA). Again, this would be a good use case for an index.
• There was a chapter that included a case study on Gboards earlier in the textbook. There is another one on Gboards in the 14.8.2 Federated Learning section -- do we want to duplicate a case study on the same topic? If not, we should probably cut this.
• 14.8.5 is titled Secure MultipartyMultiparty Communication -- we assumed this was a typo.
  Machine Learning Systems - 14 Security & Privacy.pdf

[sgiannuzzi39]

Chapter 15 - Responsible AI

• We felt like we got all Learning Objectives except the second one.
• For standardization purposes, we think that 15.2 Definition should be retitled "Terminology" in keeping with Chapter 14. However, these kinds of definitions at the beginning of the chapter really really improve readability, and we would love to see them in all chapters!
• GDPR and other regulations are defined four times in this chapter, in 15.3.1 Transparency and Explainability, 15.3.3 Privacy and Data Governance, 15.4.6 Governance, 15.6.1 Organizational and Cultural Structures and 15.5.2 Preserving Privacy . Again, this is the kind of problem an index would solve.
• 15.4 Cloud, Edge & Tiny ML is repetitive and reiterates a lot of its content several times.
• We didn't really understand the table in 15.4.1 Summary. We think this entire section should be moved to the end of 15.4 Cloud, Edge & Tiny ML so that readers are oriented in the topic before we get the summary.
• We didn't understand the graph in Figure 15.1. We recommend either replacing it or explaining it a bit more in the text.
• In 15.5.2 Preserving Privacy, the text mentions "Figure XXX" -- we assume this is a typo and the figure has not yet been included.
• In 15.6 Implementation Challenges, the text mentions Chapter 8's "Government Interventions and Regulations" -- this is not a section in Chapter 8.
• There are some in-line text errors which, as always, are included in the corresponding marked-up PDF.
  Machine Learning Systems - 15 Responsible AI.pdf

[sgiannuzzi39]

Chapter 16 - Sustainable AI

• We felt like we got all Learning Objectives for this chapter.
• We thought this chapter did a great job emphasizes the importance of the subject material, and had a clear format and structure
• This chapter all saw a tonal shift -- it had a more dramatic flair and included things like exclamation points. This tone, in our opinion, is a lot easier for students to interact with, and we would love to see it incorporated in other points in the textbook.
• We did think that the case studies were a bit too long (like 16.10 Case Study: Google's 4Ms); you could rework these by making them small "Example" sections and only included exactly relevant information. That being said, if you stick with Case Studies, their format/naming should be standardized (see feedback for Chapter 14, bullet point 5).
• We found more than a few typos in this chapter. As always, they have been marked in the attached PDF below.
• There were no images/tables to break up text from 16.5 Beyond Carbon Footprint all the way to around 16.10 Case Study: Google's 4Ms. We recommend including more visualizations to keep readers' interest.
  Chapter 16 - Sustainable AI.pdf

[sgiannuzzi39]

Chapter 17 - Robust AI

• First and foremost, this chapter was incredibly long -- nearly double the size of some of the other lengthier chapters in this book. It was so much material that it was hard for us to fully grasp what we were reading. We would recommend cutting down immensely. Some of the ways you can do that is by limiting the number of examples in 17.2 Real-World Examples (one per section would work fine), and cutting down paragraph size in the bullet point-like text in 17.4.2 Data Poisoning, 17.4.3 Distribution Shifts, 17.4.4 Detection and Mitigation, 17.5 Software Faults, and 17.6.3 Software-based Fault Injection Tools. Case studies also felt like too much to add to this already lengthy chapter, so cutting those could be beneficial.
• Again, Case Study naming/formatting should be standardized across all chapters.
• In sticking with other chapter formats, 17.2 Real-World Examples should probably be renamed "Historical Precedent." Also, the splitting into Cloud/Edge/Embedded subsections here felt unnecessary, and only worked to make the section longer.
• The figure in 17.3.1 Transient Faults does not have a number/name assigned to it.
• We felt that including a picture of a slideshow in Figure 17.13 was distracting -- perhaps cut or replace with actual image of visualization.
• We didn't understand Figure 17.15, 17.38 and 17.39.
• "Adversarial Attacks" in 17.4.1 Adversarial Attacks has already been defined earlier in the book, and does not warrant another definition. Same can be said for "Generative Adversarial Networks" in the same section.
• A lot of 17.4.2 Data Poisoning is a reiteration of Chapter 12, and therefore can be cut.

[sgiannuzzi39]

Chapter 19 - AI for Good

• Great way to wrap up the book! Easy to understand, cool examples.
• A kind of out-there idea spurred by this chapter -- we've talked about "AI for Good" and some environmental implications of AI, but what are some other ways AI is being used for negative objectives? Maybe before/after this chapter, there could be a chapter delving into how ML is being used for things like weaponry, taking away jobs creative fields, or affecting education. I'm sure there are some cool professors from the Social Studies department who might be able to write about this.
• There was some odd capitalizations and grammar mistakes in this chapter that need a look at. For example, the word "Access" or "Hunger" are continuously capitalized, and there are improper uses of dashes (-). I pointed out the ones I noticed in the PDF.
• In 19.1 Introduction, the author explains how TinyML could be used to advance the SDG goals. I'm confused as to why we're only looking at TinyML here, as opposed to how other ML systems could help forward these goals. Is it because TinyML is the newest, and therefore most likely to make advancements? If so, please explain that. Otherwise, include how other ML systems can make an impact.
• The second paragraph in 19.6 Disaster Response restates the same sentence in slightly different ways three times.
  Chapter 19 - AI For Good.pdf

[sgiannuzzi39]

Chapter 20 - Conclusion

• This is a harder chapter for me to comment on given that it's up to Vijay's discretion how much he wants to recap the book. I also think that we're going to have to rewrite the conclusion anyways once we've edited all the other chapters, as this is a reflection of what we choose to include/emphasize in each.
• The verb tense changed every other sentence in this chapter -- I would probably opt for staying in past tense, but it doesn't really matter which tense we choose so long as it's consistent.
• I feel like you can probably cut down to having only three or four sentences overviewing each chapter. Alternatively, this could be a great place to include a graph, where you highlight one takeaway for each chapter (which could manage to minimize a lot of text).
  Chapter 20 - Conclusion .pdf