
It's mostly like high level. So. So what? So what are your first thoughts on the interactive data analysis that you did?

M12  1:31:23  
I liked how fast and intuitive it is. My main concern is especially like the scatterplot, you showed me in that tutorial. If there's a lot of points and like one specific place that I want to split, then I'll probably over select, like, points in both regions, which I guess can be solved by having more rich more categories. So instead of having like Class One, two, and three, I'll have class one, 1.5, and two, and that region will be like 1.5. Okay, well, depends on the data itself at the end of the day. But that's kind of my main concern with selection. And then there's also something that I have faced in other programs where I want to make like weird shaped selections, because you only provide like squares selections now. So that's again, like I have to, instead of making like a cloud shaped selection, I'll have to sub frame that cloud into smaller, smaller squares so that I don't select from other regions. So maybe like some way of cloud, we have cheap selection tool. That would be nice. Something like a paintbrush. Exactly. Exactly. Yeah. I was thinking. So that would be my concern. But yeah, it's

Kiran Gadhave  1:32:42  
definitely possible, because so the extension doesn't really do all of this data extension just tracks the interactions from another Python library, which is way harder. And I think they got it there has support for all sorts of selections.

M12  1:32:58  
So that will be a nice addition. Like, you can even have it like, paintbrush has it like, do you want square shape or cloud we have tip for you, as a user can select, but that will increase the number of options that have which is not what was good. And, you know, another thing might be like, one good thing in Excel is kind of select all, or like they have the find function but find all and then replace all. Yes, so it can replace Yeah, but it's like replace all so here. In your second survey, it was like you only found three avalanche depth with quotation mark wrenches. So you could easily Okay, go ahead and change those three of them. But what if you had 300 out of 3000 too much work. rather have some like at that point, it be easier to code it than to do it manually. So that could be a thing. But other than that, it's way easier and faster. Like, and I don't have to care about misspelled either column names or pan the COVID stuff. Or like you missed the column here or like a quotation mark there. Just selects press buttons and done with it. I guess for like an advanced user or like an or a panda user going to this tool instead of somebody who doesn't know anything going to this tool. There will be kind of a resistance curve instead of a learning curve where they might want to see the code generated for what you're doing. Does that make sense? They'll be like, Okay, I selected this. But what's the pandas code equivalent to this? They might be but that would be a weird crowd category that you probably don't care about. And

Kiran Gadhave  1:34:53  
that's definitely one of the use cases you know, like to some interactions and generate code from your street. And then we can just run anywhere else. Yeah.

M12  1:35:05  
And they might want like, I think MATLAB does that MATLAB, you can, if you use a function that's like, find function behind it. There's like a whole C code, I think you can, like right click and see the C code that it has it shows you the code, if you want to. I don't recall ever needing to do that. But I know that exists.

Kiran Gadhave  1:35:31  
That's definitely a use case. Yeah. And get rid of just gorgeous already. But how was the learning?

M12  1:35:41  
Because you mentioned minutes. Yeah, you showed me the story, like five minutes YouTube, you would do it even even in your surveys, maybe want to do a five minute video and instead of few going through it.

Kiran Gadhave  1:35:57  
So anything, anything that was unintuitive or confusing?

M12  1:36:05  
I wouldn't say an intuitive, more like I guess this is more of what's called like the human computer interaction aspect of it, like maybe the button size, the way the table is formatted stuff like that, I guess, the HCI people might complain for me, it's fine. It's way better than just as a user's like, Okay, I'm just looking at data and I want to analyze the data and then thinking more about what's the data is exactly in the application sort of coding. So instead of my mind has to kind of make this huge jump from coding and like syntax into Okay, now I'm doing data analytics, I'm always in the data analytics mind doing data analytics, which is helpful here. Yeah. Compared to go back and forth. Okay, I want to remove the outliers. How do I do that in code? Okay, if statement and this column select and stuff.

Kiran Gadhave  1:37:03  
Again, so on that point, so usually, the workflow for removing outliers is you get a data set, you don't know there are outlets? Did you make a plot? Yeah. So what do you think about the workflow where if you're going to make a plot to look at the outliers, using the plot to then remove them as well, instead of switching to go to the motherland and creating another plot?

M12  1:37:26  
Yes, that is way easier, and especially the flow chart on the side, that's very helpful. Like if I removed too much data, or if I want to try another way of categorizing the data, then I can go back and forth between different paths on the branch. I guess. Maybe one way of like, trimming the branches would be helpful. If you have like, I like this branch, I'm working on to other branches. But this branch seems to going horribly, so I can delete it. But currently, in this situation, like I have to reset the whole thing, we do a branch and then go again, made French. Another thing that I just thought off from, I think it was the first exercise. Not sure probably easy to implement, if you haven't already implemented, but there was like a very long column. And one of the movie genres, I think it was the actual movies. So maybe a log scale button or like ceiling would make like,

Kiran Gadhave  1:38:24  
scale can definitely be switched, right. Like again, the visualization itself is like generated with the altered library, which definitely lets you select scales. It just it's hard to do it automatically without knowing because if you just switch to locksmith, and someone's not paying attention, yeah, so it just immediately, it's

M12  1:38:46  
more like, check button on the side, like, I want to see this bar chart and log scale. So it's on default, it's showing me the data assets. But then one point, there's like, one really tall bar chart that's dwarfing everything else. Like even in the last example, there was like two columns that are the outliers or maximum and the avalanche I think is key or something else. One was 88 and was one was 86. So I had to like zoom in to hover to check, like Okay, pick the 88 not the 86 one. Yeah, but so yeah, so it looks kind of sometimes makes the differences a little bit or not differences, but removes the long one, so everything else pops up more so you can see them but yeah, that's what yeah,

Kiran Gadhave  1:39:35  
that's good to know. So so now I can touch go through, you know, some of the top level tasks and I just want to know your thoughts on your date to start using this two modalities. So basically, were so for each task as we go through them. I just want your opinions on Where did you think you bench what it was? are intuitive. Which Where do you think you know versus probably is easy, but I still prefer code. So you know, for the first category we did was called a manipulation it renaming dropping columns or changing the categories

M12  1:40:17  
for changing categories, renaming, and selection, I think the preset tool was easier. Only for deletion. I might go for code. But just me.

Kiran Gadhave  1:40:31  
And why for deletion? It's

M12  1:40:34  
way easier. And like from when I searched for code, it's just delete and then the name of the column. So just easy, I guess.

Kiran Gadhave  1:40:43  
Because you can do multiple at once. Here, you have to like delete one column at a time. Yes, sir. Then like you can't delete multiple years later. Did you show that feature? Because they didn't want to have it? Yeah. Header. Can you switch to the shop? And you go to like, tutorial notebook? Yeah. So if you will click at the fourth button from top like the header? Yeah. So here, you can delete them in one go.

M12  1:41:13  
Perfect. Yeah, the preset is better. Yeah.

Kiran Gadhave  1:41:18  
I didn't check for this. Yeah, basically, I just wanted to know, you know, why would you prefer code for delete? Because I actually, can, you know, like, there is an argument to wait for rename if you have a lot of columns to rename with same pattern, maybe code. Yeah. But for delete data, like surprising. So the other alternative he did was basically, I think, covered this before, but was finding, you know, finding the data and just correcting them.

M12  1:41:54  
Yeah, so So if there's too many to correct them, probably, I'll go for code and, but if it's like in, let's say, 20 points, 15 points, then I'll correct them myself, too.

Kiran Gadhave  1:42:06  
And ideally, take a search and replace all would be perfect.

M12  1:42:11  
Yeah, that's how I do it with Excel. Now with my data when I'm working on data.

Kiran Gadhave  1:42:18  
And then for the task was you had to visit, remove subsetting remove outliers or remove

M12  1:42:24  
these error here, that the only complaint would be like, if the data is really close to each other, then I really need Python or pandas to go and like, is this less than 20 or greater than 20, like, to the third decimal point or whatever. But if the data is already kind of sparse, and like, you can see the outliers, you can see the core concentration than definitely through preset, okay. And if I'm, even if I really need like, decimal accuracy, and I'm still like, trying to learn the data, visualization will be easier through selection and moving around. So I'll probably do this. But maybe if I need more accuracy than I do the, copy that code that this generated, and then modify it to instead of my selection, for example, that like 21.25, whatever, I'll do it to 20.0. So that's maybe when the code the copying, be useful point, like if I want, or maybe even like, if you have a side text field where you take values. So I can tell you like, on the x axis, select from this site to this site. So Dr. Pan kind of the columns,

Kiran Gadhave  1:43:43  
brushing with mouse up and click ranges, what exactly it is not

M12  1:43:48  
instead, but together. So I can select and while I'm selecting, it's kind of showing you a number use that to refine

Kiran Gadhave  1:43:55  
exactly, yeah, that makes sense. Yeah, and then the last type of touch touch period was basically classifying or categorizing the data.

M12  1:44:08  
Yeah. So that's where I guess like, zooming in or using logarithm will show like, make the data more visible to me. But yeah, that way, it's easier here. I think it's the same in like, what's the difference between preset and the previous one? It was showed the bar graphs and three classes. Right. Yeah, that was the question

Kiran Gadhave  1:44:33  
that and so that actually brings me to so far the pandas one that was one of the questions just kept no good. Yeah, there. Were you struggling with time.

M12  1:44:45  
So can I go back?

Kiran Gadhave  1:44:49  
So you know, like, particular any particular action that was? It's the one before Okay, yeah. When you actually had to add the category. Yeah, I

M12  1:44:58  
think the main thing was the syntax bytes on the right, I got stuck on the syntax. Brackets, because the and this

Kiran Gadhave  1:45:06  
was also something you know, like, this was like you basically wrote this code quite, like near the end. Yeah. But before that you tried a couple of things. So what was basically what sort of things were you finding? And

M12  1:45:19  
yeah, so I, because I don't know the exact syntax for this task. The my first search online gave me like, you can do this whole thing in one sentence, which apparently, it's not, it's just, it gives you it doesn't populate the whole column properly, like, how, in your selection method, there's an selected pass. That's what the first thing I found was kind of return, which couldn't be written back to. Okay, yeah, types. So that that's why I had to search more and then find this, which gives you the full column length to the table. So that's where I got stuck.

Kiran Gadhave  1:46:02  
Okay, yeah. So because this, this way of doing it is something that I've never seen before, usually, the easiest way is, you know, you create a column and assign everything to one value override using the first method UTM across no collateral, you don't have to, you know, match the length, because there is default for everything else. I

M12  1:46:23  
guess this is worth the time constraint. Yeah, pressure, or whatever the survey. Difficulty was, if I had more time, I would have.

Kiran Gadhave  1:46:34  
Yeah, and which, you know, like, so that's something since you recursion like for right now, when you do analysis, you spend quite a bit of time looking through syntax. So what? So how often do you struggle with, there are just too many ways to do a thing in pandas or Python.

M12  1:46:55  
That's why I use MATLAB. MATLAB is easy. So it's like just one function. And usually, it's neat and works

Kiran Gadhave  1:47:06  
a lot with like numeric data,

M12  1:47:08  
I do. Yes. I do sparse matrix vector multiplication. That's one of the core math applications I work on. Probably

Kiran Gadhave  1:47:16  
like NumPy is better than Pandas for you, like NumPy is basically MATLAB,

M12  1:47:20  
probably. But the idea is kind of going back to this. Even back more backfield between the Linux camp and the windows camp, and then MATLAB and octave camps. Dave is more command line. And like more rigid, you have to do more stuff. But MATLAB since it's paid, and there's a company that supports it, not to open source, then there's like, a little bit easier for programmers. So that that's why kind of, I started with MATLAB, and from there, but then now I'm using C and C++, which is even harder than the next one. Not done, but it needs more like you have to tell it exactly what you wanted. Yeah. Otherwise, it's like, I don't know what you want to do. Yeah.

Kiran Gadhave  1:48:10  
Okay. So going back from the individual tasks, the just talked about general workflows. So. So one part of persist is you start with a data frame, and then you go to like an interactive chart or a table. And the output from that is, again, a data frame. So essentially, you know, if you run into a task, which is really tedious to do with the attraction, like renaming 100 columns, you can create a data frame, do that with code and then go back to interactions with him. So what are your thoughts on notes workflow? Is that it is why an expert isn't here. I'll just know.

Yeah, so can you just like there's a lobby sitting here, here, and I just

Yeah, so as asking about the basics of workflow where, you know, you basically use what's best for the task you want to do, and having the ability to switch, whether it's going from Excel to a Jupyter Notebook is annoying. You have to save a CSV, CSV again. Yeah. So what are your thoughts on

M12  1:49:31  
being on the same workflow, okay,

Kiran Gadhave  1:49:34  
like, you know, like alternating or switching.

M12  1:49:38  
I mean, I prefer to be in the same program instead of jumping between different programs, but sometimes, like I'm, I'm, I have been, I guess, trained or used to, like, you read data from an Excel file into MATLAB, you modify the whatever, and then spit it out into another excellent sheet. So I'm used to but I see that you point that like staying everything is in MATLAB, like you open the Excel data in MATLAB and modify it in MATLAB would be nice.

Kiran Gadhave  1:50:07  
So when that workflow is on the artifacts too much to manage all these temporary files, you have to switch. That's

M12  1:50:17  
true. Yeah, that is, like those here. Like,

Kiran Gadhave  1:50:21  
the idea is to replace those with variables which you can create when you read on the sun. Yes, you know, what sort of so like how, like, do you have like some, you know, a system to like manage this temporary file. So you just create them every time,

M12  1:50:37  
what it's the datasets I usually work on have, like too big. So even excel sheets are sometimes not big enough. So I kind of have to keep the original data set and kind of work on subsets of it. But I don't have to save like the aftermath of it, I'll just have to like plug the data or get like a statistics format, mean average and stuff like that, or eliminate, like prune the data and then work on a subset. So I'll do that every time. So there's no temp files, as much as every time I have to kind of go through the cleaning process, which is a code that's already MATLAB file that I run, and then I'll have the temporary variable that's holding everything I need. And I walk in that makes sense.

Kiran Gadhave  1:51:27  
So the next question is, would you use something like purses in your future? If,

M12  1:51:33  
if this can scale to like million by million data sets? I'll definitely use it. Yes, yeah. The only like drawback of this and I have tried multiple applications and apps that visualize data, as just when you load a million by million data set that just crashes or, like you tried to zoom in or select, like a small space to investigate more, just takes hours to rescale, remove data and show you the square root of some

Kiran Gadhave  1:52:07  
visualization techniques. So there is a new research, like it's still a recent project, but it's called Mosaic. And it basically uses the concept of data cube. So where intact, and they have a demo on the website, where they basically let you brush through and zoom into a million points. But it never really loads a million point it uses the the sexual position, you load a part of those million points. Yeah, at

M12  1:52:35  
one point, I tried to program something like this, but then, like, in 1% of the problems was like, this is going to take too much from it. So I stopped even thinking

Kiran Gadhave  1:52:46  
you should take check that. It's pretty Yeah, it's like, it's, it's basically pretty, pretty much table. You know, like that. So like, what convinced me was the demo on their website uses like a flight data set, which I know has like a million words on it. Yeah, like, Okay, this definitely looks awesome. Yeah. And it doesn't require pre processing, which is great. Yeah, yes. Yeah, I think this is pretty much what I had. Thank you. Thank you for it. 

